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Abstract 


The  Standard  CMMI®  Appraisal  Method  for  Process  Improvement  (SCAMPIsm)  A  is 
designed  to  provide  benchmark  quality  ratings  relative  to  Capability  Maturity  Modef 
Integration  (CMMI)  models.  It  is  applicable  to  a  wide  range  of  appraisal  usage  modes, 
including  both  internal  process  improvement  and  external  capability  determinations. 
SCAMPI  A  satisfies  all  of  the  Appraisal  Requirements  for  CMMI  (ARC)  requirements  for  a 
Class  A  appraisal  method. 


The  SCAMPI  vl.2  Class  A  Method  Definition  Document  describes  the  requirements, 
activities,  and  practices  associated  with  each  of  the  processes  that  compose  the  SCAMPI  A 
method.  It  is  intended  to  be  one  of  the  elements  of  the  infrastructure  within  which  SCAMPI 
Lead  AppraisersSM  conduct  a  SC  AMPI  A  appraisal.  Precise  listings  of  required  practices, 
parameters,  and  variation  limits,  as  well  as  optional  practices  and  guidance  for  enacting  the 
method,  are  covered.  An  overview  of  the  method’s  context,  concepts,  and  architecture  is  also 
provided. 
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About  This  Document 


The  Method  Definition  Document  (MDD)  describes  the  Class  A  Standard  CMMI  Appraisal 
Method  for  Process  Improvement  (SCAMPI).  It  is  intended  to  meet  the  needs  of  different 
readers.  The  document  is  divided  into  three  major  parts,  each  with  a  different  level  of  detail, 
intended  usage,  and  primary  audience.  The  structure,  audiences,  and  suggested  use  of  each 
part  of  the  document  are  described  below. 

Document  Outline 

Part  I:  Overview 

This  part  of  the  document  provides  an  overview  of  the  method’s  context,  concepts,  and 
architecture.  The  reader  is  provided  with  the  big  picture  of  the  method,  rather  than  details 
about  how  to  enact  it.  Table  I- 1  shows  the  contents  of  Part  I. 


Table  1-1:  Part  I  Contents 


Section 

Pages 

About  This  Document 

1-3  - 1-7 

Executive  Summary 

1-9 -1-13 

SCAMPI  A  Method  Overview 

1-15 -1-38 

Part  II:  Process  Definitions 

This  part  of  the  document  provides  the  definitive  coverage  of  method  requirements  and 
detailed  activities  and  practices  associated  with  each  of  the  processes  that  compose  the 
SC  AMPI  A  method.  Precise  listings  of  required  practices,  parameters,  and  limits  of  variation 
allowed,  as  well  as  optional  practices  and  guidance  for  enacting  the  method,  are  covered  in 
this  core  part  of  the  document.  Table  1-2  shows  the  contents  of  Part  II. 
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Table  1-2:  Part  II  Contents 


Phase 

Process 

Pages 

1 :  Plan  and  Prepare 
for  Appraisal 

1 . 1  Analyze  Requirements 

II-2- 11-17 

1.2  Develop  Appraisal  Plan 

11-18 -11-31 

1.3  Select  and  Prepare  Team 

11-32  - 11-44 

1 .4  Obtain  and  Inventory  Initial  Objective 
Evidence 

11-45  - 11-52 

1.5  Prepare  for  Appraisal  Conduct 

11-53  - 11-64 

2:  Conduct  Appraisal 

2. 1  Prepare  Participants 

11-65  - 11-67 

2.2  Examine  Objective  Evidence 

11-68  - 11-78 

2.3  Document  Objective  Evidence 

11-79  - 11-89 

2.4  Verify  Objective  Evidence 

11-90 -II- 100 

2.5  Validate  Preliminary  Findings 

11-101  - 11-105 

2.6  Generate  Appraisal  Results 

11-106 -11-115 

3:  Report  Results 

3 . 1  Deliver  Appraisal  Results 

11-116 -11-124 

3.2  Package  and  Archive  Appraisal  Assets 

11-125 -11-134 

Part  III:  Appendices,  References,  and  Glossary 

The  material  contained  in  the  appendices  of  this  document  provide  further  elaboration  on 
selected  topics,  and  are  intended  to  supplement  the  material  in  the  first  two  parts  of  the 
document.  Rarely  will  someone  who  has  not  already  read  the  first  two  parts  read  an  appendix 
of  this  document.  The  topical  elaboration  and  reference  material  available  in  the  appendices 
help  to  provide  deeper  insight  to  an  already  knowledgeable  reader.  Table  1-3  shows  the 
contents  of  Part  III. 


Table  1-3:  Part  III  Contents 


Section 

Pages 

Appendix  A:  Appraisal  Disclosure  Statement 

III-3  -  III- 16 

Appendix  B:  The  Role  of  Practice  Implementation  Indicators  in  Verifying 
Practice  Implementation 

III- 17  -  III-24 

Appendix  C:  Alternative  Practice  Identification  and  Characterization 
Guidance 

III-25  -  III-28 
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III-29  -  III-42 

References/Bibliography 
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Audiences  for  This  Document 


The  MDD  is  primarily  intended  for  SCAMPI  Lead  Appraisers  authorized  by  the  Software 
Engineering  Institute  (SEI).  It  is  expected  that  these  professionals  meet  prerequisites  for 
knowledge  and  skills  specified  by  the  SEI  Appraisal  program  (see  http://www.sei.cmu.edu/ 
for  details),  and  that  this  document  is  one  of  the  elements  of  the  infrastructure  within  which 
they  operate.  They  are  considered  the  primary  audience  for  Part  II.  The  MDD  is  also  used  as 
a  training  aid  in  the  SCAMPI  Lead  Appraiser  training. 


Appraisal  team  members  are  expected  to  refer  to  this  document  as  a  training  aid.  Portions  of 
the  document  may  also  be  used  as  work  aids  during  the  conduct  of  an  appraisal.  Potential 
appraisal  team  members  can  use  the  MDD  to  build  their  knowledge  base  for  future 
participation  in  an  appraisal. 


Finally,  the  stakeholders  for  the  conduct  of  any  given  appraisal  are  also  in  the  targeted 

audience  for  the  document,  particularly  for  Part  I.  These  stakeholders  include 

•  appraisal  sponsors — leaders  who  sponsor  appraisals  to  meet  business  needs 

•  Process  Group  members — process  improvement  specialists  who  need  to  understand  the 
method,  and  perhaps  help  others  to  gain  familiarity 

•  other  interested  parties  who  wish  to  have  deeper  insight  into  the  methodology  for 
purposes  such  as  ensuring  that  they  have  an  informed  basis  for  interpreting  SCAMPI  A 
outputs  or  making  comparisons  among  similar  methodologies 


How  to  Use  This  Document 

Part  I 

It  is  expected  that  every  member  of  the  audience  for  this  document  will  find  value  in  Part  I. 
The  two  primary  sections  in  this  part  are  the  Executive  Summary  and  the  Method  Overview. 


The  Executive  Summary  is  intended  to  provide  high-level  information  about  what  SCAMPI 
A  is,  and  does  not  require  extensive  knowledge  of  appraisals.  This  portion  of  the  document 
may  be  excerpted  and  provided  to  a  more  casual  reader  or  a  stakeholder  in  need  of  general 
information  to  support  their  decision  to  conduct  an  appraisal. 


The  Method  Overview  section  provides  more  comprehensive  coverage  of  SCAMPI  A,  and 
can  be  used  to  begin  building  a  base  of  knowledge  for  readers  who  have  a  need  for  more 
detailed  information.  Appraisal  sponsors  wanting  more  than  the  summary  view  described 
above  will  want  to  read  this  section.  Every  prospective  SCAMPI  A  appraisal  team  leader  and 
team  member  is  expected  to  read  this  section  of  the  document,  to  ensure  that  they  have  the 
“big  picture”  before  study  of  the  detailed  methodology  begins. 
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Part  II 


People  who  will  enact  an  appraisal  are  expected  to  read  the  second  part  of  the  document. 
Members  of  this  audience  need  to  know  how  to  enact  the  method,  not  just  what  the  method 
is.  Part  II  is  divided  into  Process  Definitions,  which  are  in  turn  divided  into  Activity 
Descriptions.  Each  Activity  Description  delineates  Required  Practices,  Parameters  and 
Limits,  Optional  Practices,  and  Implementation  Guidance. 

There  are  several  processes  contained  in  SCAMPI  A.  The  processes  (as  defined)  support  a 
variety  of  orderings  and  enactments  to  facilitate  a  variety  of  usage  modes  for  SCAMPI  A. 

The  temporal  flow,  as  well  as  the  flow  of  inputs  and  outputs  among  the  processes,  is 
described  in  the  Method  Overview  section.  The  Process  Definitions  are  not  intended  to 
provide  a  start-to-finish  view  of  SCAMPI  A.  Instead,  these  sections  provide  detailed 
definitions  of  processes  and  activities  that  are  invoked  according  to  the  appraisal  plan  created 
by  the  appraisal  team  leader. 

Each  of  the  Process  Definitions  begins  with  a  three-page  overview  of  the  process.  Every 
process  is  defined  by  information  contained  in  the  elements  shown  in  Table  1-4. 


Table  1-4:  Process  Definition  Elements 


Element 

Description 

Purpose 

A  brief  summary  of  what  is  accomplished  by  enacting  the  process 

Entry  Criteria 

Conditions  that  must  be  met  before  enacting  the  process 

Inputs 

Artifacts  or  information  needed  to  enact  the  process 

Activities 

The  set  of  actions  which,  in  combination,  make  up  the  process 

Outputs 

Artifacts  and  assets  that  result  from  enacting  the  process 

Outcome 

Any  change  in  important  conditions  or  artifacts  that  results  from 
enacting  the  process 

Exit  Criteria 

Conditions  to  be  met  before  the  process  can  be  considered  complete 

Key  Points 

A  summary  of  the  most  notable  events  associated  with  the  process 

Tools  and 
Techniques 

Work  aids  commonly  used  in  enacting  the  process 

Metrics 

Useful  measures  that  support  the  process  enactment,  or  future  enactments 

Verification  and 
Validation 

Techniques  to  verify  and/or  validate  the  enactment  of  the  process 

Records 

Information  to  be  retained  for  future  use 

Tailoring 

A  brief  discussion  of  key  tailoring  options  (not  an  exhaustive  list) 

Interfaces  with 
Other  Processes 

A  discussion  of  how  the  process  interacts  with  other  processes  in  the 
method 

Summary  of 
Activities 

A  narrative  summary  of  the  set  of  activities 
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Following  the  three  pages  of  introductory  material,  each  activity  that  is  a  part  of  the  process 
is  briefly  summarized  to  orient  the  reader  to  the  scope  of  the  activity.  Each  Activity 
Description  includes  the  elements  shown  in  Table  1-5. 


Table  1-5:  Activity  Description  Elements 


Element 

Description 

Required  Practices 

A  listing  of  practices  that  must  be  implemented  to 
consider  the  enactment  a  valid  SCAMPI  A 

Parameters  and  Limits 

Acceptable  limits  for  things  that  are  allowed  to  vary, 
and  acceptable  limits  for  things  under  the  discretion  of 
the  appraisal  team  leader 

Optional  Practices 

Actions  that  reflect  good  practice  but  are  not  required 

Implementation  Guidance 

A  narrative  description  of  advice  or  things  to  consider 
in  performing  the  activity 

Complete  and  unambiguous  descriptions  of  the  method  processes  and  activities  are  provided 
in  this  part  of  the  document.  In  combination  with  the  training  materials  and  work  aids  that 
compose  the  CMMI  Steward’s  appraisal  program,  this  information  provides  a  firm  basis  for 
standardization  (within  reasonable  limits)  of  the  practice  of  Process  Appraisals. 

Part  III 

The  appendices  of  the  document  provide  detailed  coverage  of  special  topics  as  well  as 
reference  material.  Readers  knowledgeable  in  SCAMPI  A  are  expected  to  read  these  sections 
for  further  understanding. 

Feedback  Information 

We  are  very  interested  in  your  ideas  for  improving  this  document.  See  the  CMMI  Web  site 
for  information  on  how  to  provide  feedback: 
http://www.sei.cmu.edu/cmmi/models/change-requests.html. 


If  you  have  questions,  send  an  email  to  cmmi-comments@sei.cmu.edu. 
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Executive  Summary 


What  Is  SCAMPI  A? 

The  Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPI)  is  designed  to 
provide  benchmark-quality  ratings  relative  to  Capability  Maturity  Model  Integration  (CMMI) 
models.  It  is  applicable  to  a  wide  range  of  appraisal  usage  modes,  including  both  internal 
process  improvement  and  external  capability  determinations.  SC  AMPI  A  satisfies  all  of  the 
Appraisal  Requirements  for  CMMI  (ARC)  requirements  for  a  Class  A  appraisal  method. 


SCAMPI  A  enables  a  sponsor  to 

•  gain  insight  into  an  organization’s  capability  by  identifying  the  strengths  and  weaknesses 
of  its  current  processes 

•  relate  these  strengths  and  weaknesses  to  the  CMMI  reference  model(s) 

•  prioritize  improvement  plans 

•  focus  on  improvements  (correct  weaknesses  that  generate  risks)  that  are  most  beneficial 
to  the  organization  given  its  current  level  of  organizational  maturity  or  process 
capabilities 

•  derive  capability  level  ratings  as  well  as  a  maturity  level  rating 

•  identify  development/acquisition  risks  relative  to  capability  /maturity  determinations 

As  a  Class  A  appraisal  method,  SCAMPI  A  is  an  appropriate  tool  for  benchmarking.  Sponsors 
who  want  to  compare  an  organization’s  process  improvement  achievements  with  other 
organizations  in  the  industry  may  have  a  maturity  level  determined  as  part  of  the  appraisal 
process. 

Decisions  made  on  the  basis  of  maturity  level  ratings  are  only  valid  if  the  ratings  are  based  on 
known  criteria.  Consequently,  contextual  information — organizational  scope,  reference  model 
scope,  appraisal  method  type,  the  identity  of  the  appraisal  team  leader  and  the  team — are 
items  for  which  criteria  and  guidance  are  provided  within  the  method  to  ensure  a  consistent 
interpretation  within  the  community.  Benchmarking  can  only  be  valid  when  there  is  a 
consistent  basis  for  establishing  the  benchmarks. 


The  SEI  maintains  industry  aggregates  for  appraisal  results.  These  data  are  reported  in 
industry  maturity  profiles  gathered  from  organizations  that  have  performed  appraisals  since 
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1987.  The  profile  is  based  on  appraisal  data  provided  by  SEI-trained  professionals,  and  is 
updated  twice  annually. 


As  the  CMMI  Steward,  the  SEI  supports  the  SCAMPI  A  method  and  operates  an 
authorization  program  for  SCAMPI  Lead  Appraisers.  Additional  details  can  be  found  at 
http://www.sei.cmu.edu. 

Core  Concepts  and  Approach 

SCAMPI  A,  as  a  benchmarking  appraisal  method,  relies  on  an  aggregation  of  information 
that  is  collected  via  defined  types  of  objective  evidence.  The  objective  evidence  feeds  an 
“information-processing  engine”  whose  parts  are  made  up  of  a  series  of  data  transformations. 
The  appraisal  team  observes,  hears,  and  reads  information  that  is  transformed  into  notes,  and 
then  into  characterizations  of  practice  implementation  gaps  or  compliance,  and  then  into 
preliminary  findings.  These  findings  are  validated  by  the  organizational  unit  before  they 
become  final  findings.  The  critical  concept  is  that  these  transformations  are  applied  to  data 
reflecting  the  enacted  processes  in  the  organizational  unit  and  the  CMMI  model,  and  this 
collection  of  data  forms  the  basis  for  ratings  and  other  appraisal  results. 


Planning  is  absolutely  critical  to  the  execution  of  SCAMPI  A.  All  phase  and  process  activities 
briefly  discussed  below  derive  from  a  well-articulated  plan  developed  by  the  appraisal  team 
leader  in  concert  with  members  of  the  appraised  organization  and  the  appraisal  sponsor. 

SCAMPI  A  Methodology 

SCAMPI  A  consists  of  three  phases  and  several  essential  processes,  as  was  shown  in  Table  I- 
2.  Each  phase  is  described  in  detail  below. 

Phase  1:  Plan  and  Prepare  for  Appraisal 

The  sponsor’s  objectives  for  performing  SCAMPI  A  are  determined  in  phase  1,  process  1.1, 
Analyze  Requirements.  All  other  planning,  preparation,  execution,  and  reporting  of  results 
proceed  from  this  initial  activity  according  to  the  phase  and  processes  outlined.  Because  of 
the  significant  investment  and  logistical  planning  involved,  considerable  iteration  and 
refinement  of  planning  activities  should  be  expected  in  phase  1 .  With  each  subsequent  phase, 
the  amount  of  iteration  will  decrease  as  data  are  collected,  analyzed,  refined,  and  translated 
into  findings  of  significance  relative  to  the  model. 


A  team  of  experienced  and  trained  personnel  performs  a  SCAMPI  A  over  a  period  of  time 
negotiated  by  the  sponsor  and  the  appraisal  team  leader.  The  scope  of  the  organization  to  be 
appraised,  as  well  as  the  scope  of  the  CMMI  model  (process  areas),  must  be  defined  and 
agreed  to.  The  scope  of  the  organization  and  model  provides  the  basis  on  which  to  estimate 
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personnel  time  commitments,  logistical  costs  (e.g.,  travel),  and  overall  costs  to  the  appraised 
organization  and  to  the  sponsoring  organization. 


During  the  appraisal,  the  appraisal  team  verifies  and  validates  the  objective  evidence 
provided  by  the  appraised  organization  to  identify  strengths  and  weaknesses  relative  to  the 
CMMI  model.  Objective  evidence  consists  of  documents  or  interview  results  used  as 
indicators  for  implementation  and  institutionalization  of  model  practices.  Before  the  Conduct 
Appraisal  phase  begins,  members  of  the  appraised  organization  typically  collect  and  organize 
documented  objective  evidence.  The  information-processing  “engine”  of  the  appraisal  is  thus 
fueled  by  the  objective  evidence  already  available,  saving  the  appraisal  team  the  time  and 
effort  of  a  discovery  process. 

While  it  is  not  absolutely  required  for  performance  of  a  SCAMPI  A  appraisal,  this  advance 
preparation  by  the  appraised  organization  is  key  to  the  most  efficient  execution  of  the 
method.  Analysis  of  preliminary  documented  objective  evidence  provided  by  the  appraised 
organization  plays  an  important  role  in  setting  the  stage  for  appraisal  execution.  If  substantial 
data  are  missing  at  this  point,  subsequent  appraisal  activities  can  be  delayed  or  even  cancelled 
if  the  judgment  is  made  that  continuing  appraisal  activities  will  not  be  sufficient  to  make  up 
for  the  deficiency. 


The  collection  of  documented  objective  evidence  by  the  appraised  organization  in  advance  of 
the  appraisal  not  only  improves  appraisal  team  efficiency,  but  also  offers  several  other 
benefits  to  the  organization: 

•  improved  accuracy  in  appraisal  results  delivered  by  external  appraisal  teams  (i.e.,  clear 
understanding  of  implemented  processes,  strengths,  and  weaknesses) 

•  detailed  understanding  of  how  each  project  or  support  group  has  implemented  CMMI 
model  practices,  and  the  degree  of  compliance  and  tailoring  of  organizational  standard 
processes 

•  assets  and  resources  for  monitoring  process  compliance  and  process  improvement 
progress 

•  residual  appraisal  assets  that  can  be  reused  on  subsequent  appraisals,  minimizing  the 
effort  necessary  for  preparation 


Phase  2:  Conduct  Appraisal 

In  phase  2,  the  appraisal  team  focuses  on  collecting  data  from  the  appraised  organization  to 
judge  the  extent  to  which  the  model  is  implemented.  Integral  to  this  approach  is  the  concept 
of  coverage,  which  implies  (a)  the  collection  of  sufficient  data  for  each  model  component 
within  the  CMMI  model  scope  selected  by  the  sponsor,  and  (b)  obtaining  a  representative 
sample  of  ongoing  processes  (spanning  the  lifecycle  phases  consistent  with  the  model  scope 
of  the  appraisal).  For  a  benchmarking  appraisal  methodology,  this  means  collecting  data  and 


CMU/SEI-2006-HB-002 


1-11 


information  on  all  the  CMMI  model  practices  for  each  process  instantiation  being  appraised 
within  the  organizational  unit.  The  data-collection  plan  developed  in  phase  1  undergoes 
continuous  iteration  and  refinement  until  sufficient  coverage  is  achieved. 


Upon  determining  that  sufficient  coverage  of  the  CMMI  model  and  organizational  unit  has 
been  obtained,  appraisal  findings  and  ratings  may  be  generated.  Goal  ratings  are  determined 
within  each  process  area,  which  collectively  can  be  used  to  determine  a  capability  level  rating 
for  the  individual  process  areas,  as  well  as  a  process  maturity  rating  for  the  organizational 
unit. 

Phase  3:  Report  Results 

In  phase  3,  the  appraisal  team  provides  the  findings  and  ratings  to  the  appraisal  sponsor  and 
the  organization.  These  artifacts  become  part  of  the  appraisal  record,  which  becomes 
protected  data  in  accordance  with  the  Appraisal  Disclosure  Statement.  The  level  of  protection 
and  the  plan  for  the  disposition  of  appraisal  materials  and  data  is  determined  in  phase  1  in 
collaboration  with  the  sponsor.  A  completed  appraisal  data  package,  which  includes  a  subset 
of  the  contents  of  the  appraisal  record,  is  forwarded  to  the  CMMI  Steward.  The  Steward  adds 
it  to  a  confidential  database  for  summarization  into  overall  community  maturity  and 
capability  level  profiles,  which  are  made  available  to  the  community  on  a  semiannual  basis. 

SCAMPI  A  Tailoring 

Successful  application  of  SC  AMPI  A  relies  on  adjusting  the  parameters  of  the  method  to  the 
needs  of  the  organization  and  to  the  objectives  and  constraints  of  the  sponsor’s  organization. 


The  sponsor’s  objectives  largely  influence  tailoring  decisions.  The  reference  model  scope  and 
representation  (staged  or  continuous),  the  size  of  the  organizational  unit,  the  number  and  size 
of  sampled  projects,  the  size  of  the  appraisal  team,  and  the  number  of  interviews  greatly 
influence  things  such  as  preparation  time,  time  on  site,  and  monetary  costs,  and  so  are  also 
major  factors  when  choosing  tailoring  options.  All  tailoring  decisions  must  be  documented  in 
the  appraisal  plan. 


Tailoring  should  not  exceed  the  acceptable  limits  allowed  by  the  appraisal  method.  The 
appraisal  team  leader  is  responsible  for  ensuring  that  the  requirements  of  the  method  are 
satisfied.  Tailoring  the  method  too  severely  could  result  in  the  failure  to  satisfy  method 
requirements,  the  inability  to  obtain  sufficient  data  for  generation  of  appraisal  findings  or 
ratings,  or  the  failure  to  meet  the  criteria  necessary  for  recognition  as  a  SCAMPI  A  (ARC 
Class  A)  appraisal. 
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Time  Frame  and  Personnel  Requirements 

A  requirement  of  the  SCAMPI  A  method  is  that  the  Conduct  Appraisal  phase  must  be 
completed  within  90  days.  Afterwards,  the  follow-on  activities  implicit  with  a  full  cycle  of 
appraisal  to  re-appraisal  would  include  time  for  creating  an  action  plan  and  18  to  24  months 
for  implementation,  with  a  re-appraisal  occurring  in  the  latter  6  months  of  that  period.  (The 
time  estimates  given  here  refer  to  calendar  duration  rather  than  person-months  of  effort.) 


Personnel  needed  to  participate  in  activities  or  perform  tasks  in  a  SCAMPI  A  appraisal 
include  the  sponsor,  the  appraisal  team  leader,  the  Organizational  Unit  Coordinator  (OUC), 
selected  participants,  and  appraisal  team  members.  Their  time  commitments  will  vary  a  great 
deal  depending  on  the  specific  parameters  of  the  appraisal  (e.g.,  organizational  scope)  and 
their  role. 


Typically,  appraisal  participants  can  expect  to  spend  one  to  three  hours  each  to  provide 
objective  evidence  to  the  team  and  attend  validation  sessions,  plus  one  to  three  hours  each  for 
presentations.  On  the  other  extreme,  the  OUC  may  spend  as  much  as  three  weeks  of  full  time 
effort  helping  the  team  and  the  organization  to  prepare  for  and  conduct  the  appraisal. 
Experienced  appraisal  team  leaders  will  provide  effort  estimates  corresponding  to  the  set  of 
tailoring  options  they  prefer  to  use  in  conducting  a  SCAMPI  A  appraisal. 
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SCAMPI  A  Method  Overview 


This  section  provides  an  overview  of  the  underlying  principles  and  concepts  of  the  SC  AMPI 
A  method.  Readers  of  the  SCAMPI  A  MDD  should  become  familiar  with  this  material  prior 
to  reading  the  process  descriptions  in  Part  II  of  this  document,  where  the  method 
requirements  and  tailoring  options  are  defined.  This  overview  is  primarily  targeted  at 
appraisal  team  leaders  and  appraisal  team  members  who  will  be  performing  SCAMPI  A 
appraisals.  Additional  audiences  might  include  appraisal  sponsors  or  process  improvement 
professionals  interested  in  understanding  SCAMPI  A  features  and  the  results  that  can  be 
expected. 

Method  Context 

The  SCAMPI  A  appraisal  method  is  used  to  identify  strengths,  weaknesses,  and  ratings 
relative  to  CMMI  appraisal  reference  models.  It  incorporates  best  practices  found  successful 
in  the  appraisal  community,  and  is  based  on  the  features  of  several  legacy  appraisal  methods, 
including 

•  CMM-Based  Appraisal  for  Internal  Process  Improvement  (CBA IPI)  V 1 . 1  [Dunaway 
96b]. 

•  Electronic  Industries  Alliance/Interim  Standard  (EIA/IS)  73 1 .2  Appraisal  Method  [EIA 
98b]. 

•  Software  Capability  Evaluation  (SCEsm)  V3.0  Method  Description  [Byrnes  96] 

•  Software  Development  Capability  Evaluation  (SDCE)  [AFMC  94] 

•  FAA  Appraisal  Method  (FAM)  [Ibrahim  99] 

SCAMPI  A  satisfies  the  Appraisal  Requirements  for  CMMI  (  ARC)  vl  .2  and  is  a  Class  A 
appraisal  method. 

Method  Objectives  and  Characteristics 

The  SCAMPI  A  method  has  the  following  primary  objectives: 

•  provide  a  common,  integrated  appraisal  method  capable  of  supporting  appraisals  in  the 
context  of  internal  process  improvement,  supplier  selection,  and  process  monitoring  (see 
“Modes  of  Usage”) 

•  provide  an  efficient  appraisal  method  capable  of  being  implemented  within  reasonable 
performance  constraints  (see  “Method  Performance”) 
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The  SCAMPI  A  method  is  also  designed  to  prioritize  and  satisfy  certain  essential 
characteristics,  which  were  obtained  via  community  feedback  and  are  summarized  in  Table  I- 
6.  These  characteristics  have  been  used  as  the  rationale  for  key  method  architecture  and 
design  decisions,  which  are  described  in  this  overview  and  throughout  the  MDD. 


Table  1-6:  Essential  Characteristics  of  the  SCAMPI  A  Method 


Characteristic 

Explanation 

Accuracy 

Appraisal  ratings  are  truly  reflective  of  the  organization’s 
maturity/capability,  reflect  the  appraisal  reference  model,  and  can  be 
used  for  comparison  across  organizations. 

Appraisal  results  reflect  the  strengths  and  weaknesses  of  the  appraised 
organization  (i.e.,  no  significant  strengths  and  weaknesses  are  left 
undiscovered). 

Repeatability 

The  ratings  and  findings  of  an  appraisal  are  likely  to  be  consistent  with 
those  of  another  independent  appraisal  conducted  under  comparable 
conditions  (i.e.,  another  appraisal  of  identical  scope  will  produce 
consistent  results). 

Cost/Resource 

Effectiveness 

The  appraisal  method  is  efficient  in  terms  of  person-hours  spent 
planning,  preparing,  and  executing  an  appraisal. 

The  method  takes  account  of  the  organizational  investment  in  obtaining 
the  appraisal  results,  including  the  resources  of  the  host  organization, 
the  impact  on  the  appraised  organization,  and  the  appraisal  team. 

Meaningfulness 
of  Results 

Appraisal  results  are  useful  to  the  appraisal  sponsor  in  supporting 
decision  making.  This  support  of  decision  making  may  include 
application  of  the  appraisal  results  in  the  context  of  internal  process 
improvement,  supplier  selection,  or  process  monitoring. 

ARC  Compliance 

SCAMPI  A  is  a  Class  A  method  and  complies  with  all  ARC 
requirements. 

Modes  of  Usage 

As  used  in  the  CMMI  Product  Suite  materials,  an  appraisal  is  an  examination  of  one  or  more 
processes  by  a  trained  team  of  professionals  using  an  appraisal  reference  model  as  the  basis 
for  determining  strengths  and  weaknesses.  An  appraisal  is  typically  conducted  in  the  context 
of  process  improvement  or  capability  evaluation.  The  term  “appraisal”  is  a  generic  term  used 
throughout  the  CMMI  Product  Suite  to  describe  applications  in  these  contexts,  traditionally 
known  as  assessments  and  evaluations. 


The  basic  difference  between  an  assessment  and  an  evaluation  is  that  an  assessment  is  an 
appraisal  that  an  organization  does  to  and  for  itself  for  the  purposes  of  process  improvement. 
Assessments  provide  internal  motivation  for  organizations  to  initiate  or  continue  process 
improvement  programs.  An  evaluation  is  an  appraisal  in  which  an  external  group  comes  into 
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an  organization  and  examines  its  processes  as  input  to  a  decision  regarding  future  business  or 
for  monitoring  current  business.  Evaluations  are  typically  externally  imposed  motivation  for 
organizations  to  undertake  process  improvement. 


As  an  ARC  Class  A  method,  SCAMPI  A  is  a  benchmarking-oriented  method  suitable  for 
generating  ratings.  SCAMPI  A  appraisals  can  be  performed  in  three  modes  of  usage,  as 
depicted  in  Table  1-7.  While  many  of  the  SCAMPI  A  features  are  common  across  all  usage 
modes  (e.g.,  identification  of  strengths,  weaknesses,  and  ratings),  there  are  differences  in 
motivation  and  intent  that  can  result  in  some  expected  method  differences  in  these  usage 
modes.  The  method  may  be  tailored  significantly  to  meet  the  business  objectives  of  the 
appraisal  sponsor. 


Table  1-7:  SCAMPI  A  Modes  of  Usage 


Usage  Mode 

Description 

Internal  Process 
Improvement 

Organizations  use  appraisals  to  appraise  internal  processes,  generally 
to  either  baseline  their  capability/maturity  level(s),  to  establish  or 
update  a  process  improvement  program,  or  to  measure  progress  in 
implementing  such  a  program.  Applications  include  measuring  process 
improvement  progress,  conducting  process  audits,  focusing  on  specific 
domains  or  product  lines,  appraising  specific  parts  of  the  organization, 
and  preparing  for  external  customer-led  appraisals.  In  this  manner, 
SCAMPI  A  appraisals  supplement  other  tools  for  implementing 
process  improvement  activities. 

Supplier  Selection 

Appraisal  results  are  used  as  a  high-value  discriminator  to  select 
suppliers.  The  results  are  used  in  characterizing  the  process-related  risk 
of  awarding  a  contract  to  a  supplier.  The  appraisal  results  are  typically 
only  one  criterion  among  many  used  to  select  suppliers.  Results  are 
often  used  as  a  baseline  in  subsequent  process  monitoring  with  the 
selected  supplier. 

Process  Monitoring 

Appraisal  methods  are  also  used  in  monitoring  processes  (e.g.,  after 
contract  award,  by  serving  as  input  for  an  incentive/award  fee  decision 
or  a  risk  management  plan).  The  appraisal  results  are  used  to  help  the 
sponsoring  organization  tailor  its  contract  or  process  monitoring  efforts 
by  allowing  it  to  prioritize  efforts  based  on  the  observed  strengths  and 
weaknesses  of  the  supplying  organization’s  processes.  This  usage 
mode  focuses  on  a  long-term  teaming  relationship  between  the 
sponsoring  organization  and  the  development  organization  (i.e.,  buyer 
and  supplier). 

Where  appropriate,  differences  in  the  method  requirements,  tailoring,  or  recommended 
implementation  applicable  to  these  usage  modes  are  discussed  in  process  descriptions  and 
activities  provided  in  Part  II.  These  differences  occur  most  significantly  in  the  planning 
processes  (e.g.,  appraisal  objectives,  sponsorship,  appraisal  planning,  selection  of 
participants,  and  preparation)  and  reporting  processes  (e.g.,  reporting  of  appraisal  results,  use 
of  appraisal  results  for  decision  making,  and  follow-on  activities). 
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Note  that  the  SC  AMPI  A  method  boundary  is  expressed  in  terms  of  enactment  of  the 
appraisal  method,  including  reporting  of  appraisal  results,  but  does  not  address  the  usage  of 
those  results  in  the  sponsor’s  business  context.  For  example,  the  use  of  appraisal  results  to 
identify  acquisition  risks  for  a  supplier  source  selection  is  beyond  the  scope  of  the  method. 
These  concepts  are  better  described  in  other  documentation  specific  to  those  business 
contexts,  such  as  acquisition  regulations,  standards,  and  processes. 

Method  Performance 

Performing  appraisals  efficiently  involves  minimizing  the  use  of  resources  and  the  impact  on 
appraisal  teams  and  appraised  organizations,  while  maintaining  the  essential  method 
characteristics  that  ensure  the  high  degree  of  accuracy  required  for  an  effective  benchmarking 
appraisal  method.  The  significantly  larger  size  of  the  CMMI  appraisal  reference  models 
relative  to  legacy  source  models  makes  these  tasks  an  even  greater  challenge. 


Method  performance  was  an  influential  design  driver  that  directly  resulted  in  many  SCAMPI 
A  features.  The  MDD  contains  many  recommendations  on  proven,  effective  practices  that 
contribute  positively  to  efficient  appraisals,  although  many  of  these  recommendations  may 
not  be  strict  requirements  of  the  method.  However,  the  appraisal  method  is  only  part  of  the 
solution  for  efficient  and  cost-effective  benchmarking  appraisals  capable  of  satisfying  all 
appraisal  objectives.  Appraisal  efficiency  must  also  be  a  commitment  shared  among  appraisal 
sponsors,  appraised  organizations,  and  appraisal  teams. 


Since  SCAMPI  A  is  suitable  for  benchmarking,  thus  requiring  high  confidence  in  ratings, 
thoroughness  is  necessary.  Organizations  for  which  (a)  generation  of  ratings  is  not  required, 
(b)  the  primary  application  is  identification  of  strengths  and  weaknesses  for  process 
improvement,  and  (c)  efficiency  of  appraisal  resources  is  a  primary  concern  may  be  well 
advised  to  consider  alternative  appraisal  approaches.  Their  needs  may  be  satisfied  by  less 
demanding  ARC  Class  B  or  Class  C  methods,  such  as  SCAMPI  B  or  C.  This  consideration  of 
alternatives  to  SCAMPI  A  is  particularly  true  for  organizations  that  are  early  in  their  process 
improvement  cycle.  Refer  to  “Requirements  for  CMMI  Appraisal  Method  Class  Structure” 
and  “Requirements  for  CMMI  Appraisal  Methods”  in  the  ARC  vl.2  for  further  discussion  of 
these  issues  and  for  guidance  in  selecting  an  appropriate  appraisal  method  to  fit  your  business 
objectives. 

Method  Concepts 

This  section  provides  a  description  of  fundamental  concepts  employed  by  the  SCAMPI  A 
method.  These  concepts  are  treated  here  to  provide  readers  with  an  overall  understanding  of 
the  method  prior  to  reading  the  method  Process  Definitions  in  Part  II.  Many  of  these  concepts 
are  distributed  across  several  appraisal  method  processes  or  activities,  so  it  is  important  to 
ensure  that  a  common  understanding  is  obtained  to  recognize  the  components  of  these 
concepts  as  they  appear  elsewhere  in  this  document. 
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In  addition  to  requirements  of  the  ARC,  these  method  concepts  are  derived  from,  and  heavily 
influenced  by,  the  method  objectives,  essential  method  characteristics,  appraisal  modes  of 
usage,  and  performance  objectives  described  above. 


Method  Assumptions  and  Design  Principles 

In  addition  to  the  factors  just  mentioned,  SCAMPI  A  features  are  based  on  certain  method 
assumptions  and  design  principles  related  to  the  expected  use  of  the  method.  Those 
assumptions  and  principles  are  described  below. 


SCAMPI  A  is  a  Class  A  benchmarking  method. 

As  an  ARC  Class  A  method,  SCAMPI  A  can  be  used  to  generate  ratings  as  benchmarks  to 
compare  maturity  levels  or  capability  levels  across  organizations.  SCAMPI  A  is  an  integrated 
appraisal  method  that  can  be  applied  in  the  context  of  internal  process  improvement,  supplier 
selection,  or  process  monitoring.  As  a  benchmarking  method,  the  SCAMPI  A  emphasis  is  on 
a  rigorous  method  capable  of  achieving  high  accuracy  and  reliability  of  appraisal  results 
through  the  collection  of  objective  evidence  from  multiple  sources. 


Goal  ratings  are  a  function  of  the  extent  to  which  the  corresponding  practices  are  present  in 
the  planned  and  implemented  processes  of  the  organization. 

In  the  CMMI  appraisal  reference  models,  there  is  a  direct  relationship  between  goals  (specific 
and  generic)  and  the  practices  (specific  and  generic)  that  contribute  toward  achievement  of 
those  goals.  Specific  and  generic  goals  are  required  model  components;  specific  and  generic 
practices  are  expected  model  components  in  that  alternative  practices  could  be  implemented 
that  are  equally  effective  in  achieving  the  intent  of  the  associated  goals. 


In  the  SC  AMPI  A  method,  a  fundamental  premise  is  that  satisfaction  of  goals  can  be 
determined  only  upon  detailed  investigation  of  the  extent  to  which  each  corresponding 
practice  is  implemented  for  each  sample  instance  used  as  a  basis  for  the  appraisal  (e.g.,  each 
project). 

Additional  information  on  rating  goals  is  provided  in  “Data  Collection,  Rating,  and 
Reporting”  on  page  1-25. 


The  aggregate  of  objective  evidence  provided  is  used  as  the  basis  for  determining  practice 
implementation. 

To  make  reasonable  judgments  regarding  an  organization’s  implemented  processes  relative  to 
the  appraisal  reference  model,  appraisal  teams  base  their  judgments  on  the  collection  of 
objective  evidence  for  each  specific  and  generic  practice  applicable  to  process  area  goals 
within  the  appraisal  scope. 
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Appraisal  teams  compare  the  objective  evidence  collected  against  the  corresponding  practices 
in  the  appraisal  reference  model.  In  making  inferences  about  the  extent  to  which  practices  are 
or  are  not  implemented,  appraisal  teams  draw  on  the  entire  model  document  to  understand  the 
intent  of  the  model,  and  use  it  as  the  basis  for  their  decisions.  This  comparison  includes  the 
required  and  expected  model  components  (i.e.,  generic  and  specific  goals,  generic  and 
specific  practices)  as  well  as  informative  material,  such  as  model  front  matter,  introductory 
text,  glossary  definitions,  and  subpractices. 


Practice  implementation  at  the  organizational  unit  level  is  a  function  of  the  degree  of 
practice  implementation  at  the  instantiation  level. 

Practices  described  in  the  CMMI  appraisal  reference  models  are  abstractions  that  are  realized 
by  their  implementation  within  organizations,  and  instantiated  at  the  level  of  projects  and 
support  groups.  The  context  within  which  the  practice  is  applied  drives  the  implementation. 
The  details  of  the  implementation,  as  well  as  the  context  within  which  the  practice  is 
implemented,  are  referred  to  as  the  instantiation  of  the  practice. 

An  organizational  unit  is  the  part  of  an  organization  that  is  the  focus  of  an  appraisal.  An 
organizational  unit  operates  within  a  coherent  process  context  and  a  coherent  set  of  business 
objectives.  It  may  consist  of  a  set  of  related  projects.  (Refer  to  the  glossary  for  a  complete 
definition  of  organizational  unit.) 


The  extent  to  which  an  organizational  unit  has  implemented  appraisal  reference  model 
practices  can  be  determined  only  by  considering,  in  aggregate,  the  extent  to  which  those 
practices  are  implemented  within  the  organizational  unit  by  projects  and  support  groups.  This 
process,  in  turn,  necessitates  the  consideration  of  objective  evidence  for  each  instantiation, 
for  each  model  practice  within  the  appraisal  scope. 


Appraisal  teams  are  obligated  to  seek  and  consider  objective  evidence  of  multiple  types  in 
determining  practice  implementation  and  goal  satisfaction. 

The  SCAMPI  A  method  is  data  oriented  in  that  decisions  on  practice  implementation  and 
goal  rating  are  made  based  on  the  aggregate  of  objective  evidence  available  to  the  appraisal 
team.  Multiple  types  of  objective  evidence  must  be  considered;  these  types  are  described  in 
“Types  of  Objective  Evidence”  on  page  1-22.  Artifacts  indicative  of  practice  implementation 
are  a  requirement  of  the  SC  AMPI  A  method.  Oral  affirmations  (see  Table  1-8)  are  required  to 
ensure  that  the  documentation  reflects  the  actual  organizational  process  implementation  and 
to  preclude  rating  judgments  being  made  solely  on  the  basis  of  artifacts.  The  SCAMPI  A 
method  establishes  minimum  requirements,  described  in  activity  2.4.1,  “Verify  Objective 
Evidence,”  for  the  extent  to  which  objective  evidence  from  oral  affirmations  must  be 
collected  for  model  practices  to  corroborate  other  sources  of  objective  evidence  prior  to 
rating  goals. 
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Verification  Versus  Discovery 

If  an  organization  has  in  place  assets,  mechanisms,  and  objective  evidence  that  substantiate 
its  implementation  of  model  practices,  it  is  in  the  organization’s  best  interest  to  share  that 
knowledge  to  ensure  that  the  appraisal  team  obtains  a  complete  and  accurate  understanding  of 
the  organization’s  implementation  of  model  practices.  Many  organizations  support  this 
understanding  through  assets  such  as  traceability  and  mapping  tables  from  the  model  to  their 
processes  and  the  associated  documentation  (e.g.,  policies,  project  plans,  and  implementation 
artifacts).  Implementation  of  the  model  within  the  organization  may  be  further  reinforced 
through  additional  mechanisms,  such  as 

•  verification  and  oversight  activities  (e.g.,  internal  appraisals,  audits,  reviews,  and  status 
reports) 

•  tools  and  resources  (e.g.,  databases,  measurement  repositories,  and  configuration 
management  tools) 

Some  legacy  appraisal  methods  encouraged  organizations  to  provide  such  assets.  If  assets 
such  as  these,  or  indicators  of  the  existence  of  the  assets,  are  made  available  to  the  appraisal 
team,  this  approach  leaves  the  appraisal  team  the  task  of  verifying  whether  the  objective 
evidence  provided  is  adequate  for  substantiation  of  practice  implementation.  This 
verification-based  approach  is  in  contrast  to  the  more  difficult,  error  prone,  and  time- 
consuming  task  of  investigating  each  practice  to  discover  the  objective  evidence  needed  to 
substantiate  implementation. 


In  a  verification-based  approach,  both  the  organizational  unit  and  the  appraisal  team  have  a 
clearer  picture  of  what  artifacts  are  available  and  what  might  still  be  needed,  thereby 
minimizing  the  amount  of  further  investigation  necessary  in  the  form  of  interviews  and 
additional  documentation  requests.  The  verification-based  approach  thus  facilitates  appraisals 
that  are  accurate,  repeatable,  efficient,  and  that  provide  meaningful  results;  in  other  words, 
appraisals  that  satisfy  the  essential  method  characteristics  described  in  “Method  Objectives 
and  Characteristics”  on  page  1-15. 


The  SCAMPI  A  method  is  designed  on  the  assumption — although  not  the  requirement — that 
relevant  documented  objective  evidence  is  available  for  review  in  advance.  This  assumption 
is  typically  discussed  with  the  appraisal  sponsor  and  his/her  representatives  during 
development  of  the  appraisal  plan,  and  a  decision  is  made  whether  to  conduct  a  verification- 
based  or  a  discovery-based  appraisal. 


The  decision  to  conduct  a  verification-based  appraisal  is  revisited  at  a  key  point  in  the 
appraisal  process  when  a  review  prior  to  the  Conduct  Appraisal  phase  is  performed  to 
determine  readiness  to  proceed  with  such  an  appraisal  as  planned.  If  the  appraised 
organization  has  not  provided  documented  objective  evidence  of  sufficient  quality  and 
completeness  to  enable  a  verification-based  appraisal,  the  appraisal  plan  should  be  updated  to 
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reflect  the  effort  that  must  be  undertaken  for  the  appraisal  team  to  search  for  and  discover 
that  objective  evidence  during  the  Conduct  Appraisal  phase.  Even  when  the  readiness  review 
determines  that  a  verification-based  appraisal  can  go  forward,  some  discovery  of  additional 
documented  objective  evidence  will  typically  occur  in  the  form  of  additional  documentation 
requests  based  on  interview  results. 

Types  of  Objective  Evidence 

The  SCAMPI  A  method  provides  for  the  collection  and  analysis  of  data  from  the  following 
types  of  objective  evidence: 

•  documents  -  written  information  relative  to  the  implementation  of  one  or  more  model 
practices.  These  documents  may  include  organizational  policies,  procedures, 
implementation-level  artifacts,  instruments  (e.g.,  questionnaires),  and  presentation 
materials.  Documents  may  be  available  in  hardcopy  or  softcopy  or  accessible  via 
hyperlinks  in  a  Web-based  environment. 

•  interviews  -  oral  interaction  with  those  implementing  or  using  the  processes  within  the 
organizational  unit.  Interviews  are  typically  held  with  various  groups  or  individuals,  such 
as  project  leaders,  managers,  and  practitioners.  A  combination  of  formal  and  informal 
interviews  may  be  held,  using  interview  scripts  or  exploratory  questions  developed  to 
elicit  the  information  needed.  A  presentation  or  demonstration  may  serve  as  an  interview 
if  interaction  between  the  appraisal  team  and  presenter  can  ensue. 


Using  multiple  data-gathering  mechanisms  improves  the  depth  of  understanding  and  enables 
corroboration  of  the  data. 


Focused  Investigation 

Due  to  the  quantity  of  CMMI  appraisal  reference  model  practices  that  must  be  investigated 
and  the  SCAMPI  A  rules  for  collection  of  objective  evidence  to  ensure  sufficient  coverage  of 
these  practices  for  rating  (see  “Data  Collection,  Rating,  and  Reporting”  on  page  1-25),  it  is 
crucial  that  appraisal  teams  apply  efficient  techniques  for  the  collection  and  management  of 
appraisal  data.  This  focus  on  efficient  data  management  practices  is  integral  to  SCAMPI  A 
method  concepts,  and  is  emphasized  throughout  the  appraisal  process.  The  term  “focused 
investigation”  is  used  in  SCAMPI  A  to  describe  this  concept  of  optimized  investment  of 
appraisal  resources.  Essentially,  this  approach  can  be  described  at  a  top  level  using  the 
following  data  collection  and  investigation  paradigms: 

•  Understand  what  objective  evidence  is  available,  and  how  it  contributes  toward 
implementation  of  model  practices  within  the  appraisal  scope. 

•  Continually  consolidate  data  to  determine  progress  toward  sufficient  coverage  of  model 
practices. 

•  Focus  appraisal  resources  by  targeting  those  areas  for  which  further  investigation  is 
needed  to  collect  additional  data  or  verify  the  set  of  objective  evidence. 
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•  Avoid  unnecessary  or  duplicated  effort  that  does  not  contribute  additional  information 
toward  achievement  of  sufficient  coverage  or  toward  obtaining  significantly  greater 
confidence  in  the  appraisal  results.  For  example,  keep  interviews  efficient  by  asking 
further  questions  only  about  practices  for  which  sufficient  data  has  not  already  been 
obtained. 


These  concepts,  derived  from  the  best  practices  of  experienced  appraisal  team  leaders,  are  the 
primary  mechanisms  used  to  achieve  efficient  appraisal  performance  by  emphasizing  the 
placement  of  appraisal  team  effort  where  it  is  most  needed.  This  approach  begins  with  the 
initial  collection  and  analysis  of  objective  evidence  from  the  organizational  unit.  This 
analysis  can  be  used  to  determine  the  adequacy  and  completeness  of  the  provided  objective 
evidence,  and  to  identify  the  extent  to  which  further  investigation  is  necessary.  The  appraisal 
team’s  inventory  of  objective  evidence  can  be  annotated  to  identify  practices  that  are  strongly 
supported,  or  those  that  need  further  clarification.  This  knowledge  can  be  used  as  the  basis 
for  determining  findings  that  affect  appraisal  outcomes. 


As  the  appraisal  process  progresses,  the  appraisal  team  aggregates  and  synthesizes  additional 
objective  evidence,  and  uses  this  evidence  to  draw  inferences  about  the  overall 
implementation  within  the  organizational  unit.  Wherever  there  are  shortcomings  in  the 
appraisal  team’s  understanding  of  the  organizational  unit’s  implementation  of  model 
practices,  data-collection  strategies  can  be  determined  to  probe  for  and  obtain  additional 
information.  For  example,  cases  where  the  objective  evidence  is  missing,  unclear,  or 
insufficient  might  be  addressed  through  additional  documentation  requests  or  by  generating 
focused  questions  for  specific  interview  participants.  By  maintaining  a  current  inventory  of 
the  status  of  the  appraisal  objective  evidence  and  prioritizing  areas  where  additional 
information  is  still  needed,  these  focused  investigation  approaches  can  be  continuously  and 
iteratively  applied  to  narrow  remaining  gaps  and  converge  on  sufficient  coverage  for 
proceeding  with  rating. 

Practice  Implementation  Indicators 

The  fundamental  idea  of  Practice  Implementation  Indicators  (PIIs)  is  that  the  conduct  of  an 
activity  or  the  implementation  of  a  practice  results  in  “footprints”  (i.e.,  evidence  that  provides 
a  basis  for  verification  of  the  activity  or  practice). 


In  SCAMPI  A,  PIIs  are  the  necessary  or  incidental  consequence  of  implementing  reference 
model  practices.  For  example,  the  establishment  of  an  artifact,  such  as  a  document,  is  often 
an  expected  outcome  resulting  from  implementation  of  a  model  practice.  Other  indicators 
may  indirectly  substantiate  implementation  of  the  practice,  such  as  evidence  of  a  status 
meeting  or  peer  review  being  held.  Members  of  the  organizational  unit  may  affirm  through 
questionnaires  or  interviews  that  the  practice  is  implemented.  These  indicators  are  all 
potential  “footprints”  that  can  be  used  as  objective  evidence  to  verify  and  substantiate 
implementation  of  model  practices. 
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SCAMPI  A  characterizes  PIIs  according  to  the  indicator  types  described  in  Table  1-8. 


Table  1-8:  Practice  Implementation  Indicator  Types 


Indicator  Type 

Description 

Examples 

Direct  artifacts 

The  tangible  outputs  resulting  directly 
from  implementation  of  a  specific  or 
generic  practice.  An  integral  part  of 
verifying  practice  implementation.  May 
be  explicitly  stated  or  implied  by  the 
practice  statement  or  associated 
informative  material. 

Typical  work  products  listed 
in  reference  model  practices 

Target  products  of  an 
“Establish  and  Maintain” 
specific  practice 

Documents,  deliverable 
products,  training  materials, 
etc. 

Indirect  artifacts 

Artifacts  that  are  a  consequence  of 
performing  a  specific  or  generic  practice 
or  that  substantiate  its  implementation, 
but  which  are  not  the  purpose  for  which 
the  practice  is  performed.  This  indicator 
type  is  especially  useful  when  there  may 
be  doubts  about  whether  the  intent  of  the 
practice  has  been  met  (e.g.,  an  artifact 
exists  but  there  is  no  indication  of  where 
it  came  from,  who  worked  to  develop  it, 
or  how  it  is  used). 

Typical  work  products  listed 
in  reference  model  practices 

Meeting  minutes,  review 
results,  status  reports, 
presentations,  etc. 

Performance  measures 

Affirmations 

Oral  or  written  statements  confirming  or 
supporting  implementation  (or  lack  of 
implementation)  of  a  specific  or  generic 
practice.  These  statements  are  usually 
provided  by  the  implemented  of  the 
practice  and/or  internal  or  external 
customers,  but  may  also  include  other 
stakeholders  (e.g.,  managers  and 
suppliers). 

Instruments 

Interviews 

Presentations, 
demonstrations,  etc. 

Appraisal  teams  are  obligated  to  seek  objective  evidence  as  a  prerequisite  to  formulating 
characterizations  of  practice  implementation.  The  indicator  types  that  will  be  most 
appropriate  to  reflect  practice  implementation  will  vary  according  to  the  context  in  which  the 
process  is  implemented,  as  well  as  the  practice  itself.  The  appraisal  team  should  consider  all 
aspects  of  the  process  context,  such  as  project  size  and  duration,  organizational  culture, 
application  domain,  customer  market,  and  so  on,  in  determining  the  appropriateness  and 
sufficiency  of  indicators.  For  example,  the  level  of  detail  necessary  for  a  work  breakdown 
structure  will  differ  widely  for  a  1  -person,  2-week  maintenance  effort  as  opposed  to  a  1 00- 
person,  multi-year,  mission-critical,  new  product  development. 
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An  example  of  how  PIIs  for  each  of  these  indicator  types  might  be  used  in  verifying  the 
implementation  of  a  model  practice  is  depicted  in  Figure  I- 1 . 


PP  SP1.1  Indirect  work  product: 


Indirect  work  product:  Affirmation: 

Project  estimates  aligned  How  is  the  WBS  used? 

with  WBS  elements  How  are  estimates  generated? 


Figure  1-1:  Exam  pie  of  Pll  Use 

Appraisal  teams  collect  and  organize  data  according  to  these  indicator  types.  The  SCAMPI  A 
method  defines  rules  and  guidelines  (described  in  “Data  Collection,  Rating,  and  Reporting” 
below)  about  the  amount  of  data  that  must  be  collected  for  each  of  these  indicator  types.  A 
combination  of  objective  evidence  according  to  these  indicator  types  is  necessary  to 
corroborate  multiple  sources  of  data  that  may  be  available  for  each  practice,  and  to  obtain 
confidence  in  the  accuracy  of  the  data  collected.  For  reasons  that  are  evident,  an  over-reliance 
on  one  type  of  objective  evidence  or  another  is  undesirable.  Too  much  dependence  on 
artifacts  could  result  in  the  perception  that  the  appraisal  was  a  “paper  review”  and  not  truly 
indicative  of  organizational  and/or  project  behavior.  An  over-reliance  on  affirmations  could 
be  criticized  as  not  truly  objective  or  repeatable.  Therefore,  the  SCAMPI  A  method  requires  a 
balance  across  these  types  of  objective  evidence. 


Appendix  B  contains  additional  detailed  discussion  of  PIIs  and  indicator-based  appraisals. 

Data  Collection,  Rating,  and  Reporting 

The  appraisal  team  follows  a  consensus-based,  structured  process  to  synthesize  and  transform 
information  collected  from  the  sources  described  in  “Types  of  Objective  Evidence”  on  page 
1-22.  Data  from  these  sources  are  collected  and  considered  in  several  discrete  data-gathering 
sessions,  either  as  integrated  appraisal  team  activities  or  by  subsets  of  the  team  organized  into 
mini-teams  operating  in  parallel.  Mini-teams  are  typically  organized  around  related  process 
areas,  with  mini-team  members  assigned  by  the  appraisal  team  leader  on  the  basis  of  their 
individual  experience,  knowledge,  and  skills. 
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The  SCAMPI  A  data  transformation  and  rating  process  is  depicted  in  Figure  1-2. 


Level  of 
Consensus 


Capability  Level  and/or 
Maturity  Level  Ratings 


Full  Team 


Goal  Satisfaction  Ratings 


Full  Team 


Practice  Implementation  Characterizations 
(organizational  unit  level) 


Full  Team 


Practice  Implementation  Characterizations 
(practice  instantiation  level) 


Mini-Team 


Figure  1-2:  SCAMPI  A  Rating  Process 

Team  members  review  objective  evidence  provided  by  the  organizational  unit  and  identify 
PIIs  relative  to  the  appraisal  reference  model  practices.  These  PIIs  are  categorized  as  direct 
artifacts,  indirect  artifacts,  or  affirmations,  as  described  in  “Practice  Implementation 
Indicators”  on  page  1-23,  and  are  added  to  the  team’s  PII  inventory. 


Areas  of  significant  strength  or  weakness  observed  relative  to  the  implementation  of  model 
specific  or  generic  practices  are  recorded  in  written  findings.  Findings  are  generated 
primarily  for  weaknesses,  or  “gaps,”  of  the  implementation  compared  to  the  intent  of  a  model 
practice.  Findings  of  strengths  should  be  reserved  for  implemented  practices  that  are 
particularly  effective  and  are  candidates  for  inclusion  in  aggregated  findings.  Gratuitous 
strengths  that  simply  reflect  a  sufficient  implementation  of  a  practice  can  produce  substantial 
data  management  overhead  that  does  not  contribute  toward  generation  of  findings;  these 
gratuitous  strengths  are  more  effectively  captured  as  indicators  in  the  appraisal  team’s  PII 
inventory.  Findings  may  also  be  generated  for  alternative  practices,  which  are  acceptable 
alternatives  to  implementing  one  or  more  model  practices  that  contribute  equivalently  to  the 
satisfaction  of  process  area  goals. 

Characterizing  Practices 

Verification  of  PIIs  continues  in  this  way  at  the  instantiation  level  until  sufficient  objective 
evidence  has  been  obtained  to  characterize  the  implementation  of  a  specific  or  generic 
practice.  Sufficiency  at  the  practice  level  is  determined  when  direct  artifacts  covering  the 
intent  of  the  practice  have  been  verified  for  the  practice  and  corroborated  by  indirect  artifacts 
or  affirmations.  While  written  affirmations  can  be  used  to  confirm  implementation  of 
practices  during  the  characterization  activity  described  above,  oral  affirmations  are  still 
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required  to  the  extent  described  in  activity  2.4.1,  “Verify  Objective  Evidence.”  Consensus  is 
obtained  at  the  mini-team  level  on  the  sufficiency  of  PIIs  and  the  accuracy  of  strengths  and 
weaknesses. 

Based  on  the  practice  implementation  data,  the  appraisal  team  (or  typically  a  mini-team) 
assigns  values  to  characterize  the  extent  to  which  the  appraisal  reference  model  practice  is 
implemented.  Each  practice  is  characterized  as  Fully  Implemented  (FI),  Largely  Implemented 
(LI),  Partially  Implemented  (PI),  Not  Implemented  (NI),  or  Not  Yet  (NY). 


The  intent  of  this  characterization  is  to  summarize  the  appraisal  team’s  judgment  of  practice 
implementation  so  as  to  enable  the  identification  and  prioritization  of  areas  where  further 
judgment,  investigation,  or  corroboration  may  be  necessary.  These  characterization  values  are 
an  aid,  not  a  replacement,  for  the  recorded  findings  of  weaknesses,  which  are  used  as  a  basis 
for  rating  decisions. 


Upon  assigning  characterization  values  for  a  given  model  practice  for  each  instantiation,  the 
characterization  values  are  aggregated,  using  full  appraisal  team  consensus,  to  the 
organizational  unit  level.  Weaknesses  across  the  projects  and  support  groups  are  similarly 
aggregated  to  the  organizational  unit  level,  and  form  the  basis  for  rating.  Where  team 
judgment  is  necessary  to  characterize  practice  implementation,  these  decisions  are  made 
considering  factors  such  as  the  mix  of  practice  characterizations,  the  reason  for  the 
instantiation-level  characterizations,  and  the  severity  of  the  associated  weaknesses  (in 
aggregate). 

Tracking  Progress 

The  appraisal  team  uses  focused  investigation  techniques  (see  “Focused  Investigation”  on 
page  1-22)  to  track  progress  toward  sufficient  coverage  necessary  for  rating  process  area 
goals  within  the  appraisal  scope.  Revisions  to  the  data-collection  plan  may  be  necessary  to 
ensure  that  adequate  objective  evidence  is  obtained  for  each  specific  and  generic  practice 
within  the  reference  model  scope  of  the  appraisal.  If  insufficient  objective  evidence  is 
available,  the  data  collection  plan  may  be  revised  to  conduct  additional  data-gathering 
sessions.  Focused  investigation  techniques  can  be  used  to  ensure  progress  toward  sufficient 
coverage  of  model  practices,  goals,  and  process  areas  within  the  appraisal  scope. 

Generating  Findings 

Strengths  and  weaknesses  identified  across  projects  and  support  groups  within  the 
organizational  unit  are  synthesized  and  aggregated  to  statements  of  preliminary  findings, 
expressed  at  the  organizational  unit  level.  These  strengths  and  weaknesses  are  often 
organized  at  the  level  of  process  area  goals  using  common  themes.  Preliminary  findings  are 
provided  to  the  organizational  unit  for  validation;  the  mechanisms  and  timeframe  used  for 
this  validation  may  vary  across  the  appraisal  modes  of  usage  (internal  process  improvement, 
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supplier  selection,  process  monitoring).  During  this  activity,  the  appraisal  team  is  still  in  the 
process  of  collecting  data  to  ensure  that  an  accurate  understanding  of  the  organizational 
process  implementation  is  obtained.  Feedback  from  the  participants  in  the  appraisal  is  used  to 
validate  the  preliminary  findings  and  may  result  in  revised  or  additional  findings.  The 
appraisal  team  may  also  request  additional  data  sources  for  areas  where  their  understanding 
of  the  organization’s  implementation  of  model  practices  is  insufficient.  Final  findings  are 
generated  based  on  the  complete,  validated  set  of  appraisal  data  (i.e.,  findings  and  additional 
aggregated  strengths  and  weaknesses,  if  any). 

Generating  Ratings 

Ratings  are  generated  based  on  the  set  of  validated  appraisal  data.  At  a  minimum,  ratings  are 
generated  for  each  of  the  process  area  generic  and  specific  goals  within  the  appraisal 
reference  model  scope.  Ratings  may  also  be  generated  for  process  areas,  capability  levels,  or 
maturity  levels  if  desired  by  the  appraisal  sponsor.  Maturity  level  ratings  and/or  capability 
level  ratings  are  based  on  the  definitions  of  capability  levels  and  maturity  levels  in  the  CMMI 
appraisal  reference  model.  Refer  to  Process  Description  2.4,  “Generate  Appraisal  Results,” 
for  additional  information  about  SCAMPI  A  rating  processes. 

Reporting  Results 

The  results  of  the  appraisal  are  reported  to  the  appraisal  sponsor  and  to  the  appraised 
organization.  In  supplier  selection  or  process  monitoring  contexts,  the  mechanisms  and 
timeframe  used  for  reporting  results  may  be  subject  to  acquisition  or  contractual  restrictions. 
An  appraisal  record  is  generated  and  provided  to  the  sponsor,  documenting  further 
information  regarding  the  appraisal  according  to  the  specifications  found  in  later  sections  of 
this  document.  Appraisal  results  are  valid  for  a  period  not  to  exceed  three  years. 


A  subset  of  this  data  is  provided  to  the  CMMI  Steward  for  the  purposes  of  quality  control  and 
the  collection  of  appraisal  measures  for  reporting  to  the  appraisal  community.  The  appraisal 
data  to  be  provided  is  defined  by  the  Steward  separately  from  this  document  to  allow  for 
continuous  improvement  of  appraisal  reporting  apart  from  the  CMMI  Product  Suite. 

Instruments  and  Tools 

Instruments  are  artifacts  that  may  be  used  in  an  appraisal  for  the  collection  and  presentation 
of  data.  Instruments  are  provided  by  the  organizational  unit  to  inform  the  appraisal  team 
about  the  processes  implemented  in  the  organization  and  how  they  relate  to  the  appraisal 
reference  model.  Instruments  can  take  various  forms,  including  questionnaires,  surveys,  site 
orientation  packets,  and  mappings  from  reference  model  practices  to  the  organizational  or 
project  processes. 
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The  SCAMPI  A  method  does  not  require  that  an  instrument  be  used.  However,  instruments 
can  provide  the  appraisal  team  with  an  in-depth  understanding  of  the  organizational 
implementation  of  the  model  on  a  practice-level  basis  for  the  projects  and  support  groups 
within  the  organizational  unit  to  be  investigated  in  the  appraisal.  Instruments  also  often 
provide  an  opportunity  for  the  organizational  unit  to  provide  a  self-characterization  of  their 
implemented  processes,  identify  applicable  substantiating  objective  evidence,  and  specify  any 
additional  comments  that  might  be  useful  in  understanding  the  implemented  processes.  Used 
in  this  manner,  instruments  can  support  the  SCAMPI  A  method  emphasis  on  verification- 
based  appraisals  and  minimize  the  need  for  discovery  of  objective  evidence  (see  “Verification 
Versus  Discovery”  on  page  1-21),  thus  helping  to  facilitate  efficient  appraisal  performance. 


As  described  in  “Practice  Implementation  Indicators”  on  page  1-23,  the  SCAMPI  A  method 
emphasizes  the  use  of  PIIs.  Organizations  may  provide  as  input  to  the  appraisal  a  PII 
database  (PIIDB),  with  a  mapping  of  model  practices  to  corresponding  processes  and 
objective  evidence  that  can  be  used  to  verify  practice  implementation.  Many  organizations 
will  have  existing  assets  in  place  that  reflect  their  process  implementation  and  mapping  to 
appraisal  reference  model  practices.  These  instruments  can  be  used  as  a  source  of  appraisal 
data  in  much  the  same  way  as  a  PIIDB.  The  collection  of  these  model  mappings  and 
indicators  can  be  a  valuable  resource  for  process  improvement  at  the  organization  and  project 
levels,  and  a  rich  source  of  data  for  process  appraisals  using  a  variety  of  Class  A,  B,  and  C 
appraisal  methods. 


It  is  recommended  that  a  member  of  the  appraisal  team  facilitate  the  entry  of  data  into 
instruments  where  feasible  to  ensure  that  appropriate  data  are  obtained.  This  approach  can 
help  the  appraised  organization  clarify  or  interpret  the  intent  of  the  model  practices, 
understand  what  data  are  expected,  and  focus  the  responses.  The  entry  of  either  too  much  or 
too  little  data  into  instruments  can  be  problematic  for  both  the  appraisal  team  and  the 
appraised  organization  and  result  in  inefficient  use  of  resources. 


Effective  management  of  appraisal  data  is  a  significant  challenge  that  can  be  simplified  with 
the  use  of  automated  tools.  The  CMMI  Steward  provides  a  rudimentary  toolkit  to  SCAMPI 
Lead  Appraisers  that  can  be  used  to  collect  practice-level  data  and  to  characterize, 
consolidate,  and  summarize  information.  Several  vendor  tools  are  also  available  in  the 
marketplace.  The  choice  of  tools  is  largely  one  of  personal  preference;  some  experienced 
appraisal  team  leaders  prefer  manual  techniques,  such  as  wall  charts,  to  record  information 
and  findings. 

Effective  Team  Practices 

Appraisal  team  dynamics  and  effective  group  techniques  contribute  significantly  to  the  ability 
to  conduct  SCAMPI  A  appraisals.  The  appraisal  team  leader  can  help  focus  team  activities  so 
that  effort  is  spent  wisely  toward  achievement  of  method  requirements  and  appraisal 


CMU/SEI-2006-HB-002 


1-29 


objectives.  SCAMPI  A  features  encourage  effective  team  practices  that,  with  the  support  of 
the  appraisal  team  leader,  can  address  some  of  the  issues  that  can  impact  appraisal 
performance.  These  features  include 


•  virtual  meeting  technology >  -  Virtual  methods  such  as  video  conferences,  teleconferences, 
and  other  similar  technologies  can  be  used  to  help  reduce  the  travel  needed  to  perform  an 
appraisal.  However,  these  methods  need  to  be  clearly  defined  in  the  Appraisal  Plan. 
Furthermore,  it  is  the  responsibility  of  the  appraisal  team  leader  to  ensure  that  the  use  of 
virtual  methods  does  not  compromise  the  integrity  or  accuracy  of  the  appraisal  activities 
or  the  appraisal  results.  Virtual  methods  should  allow  for  adequate  interaction  between 
the  appraisal  team  members  and  the  appraisal  participants  and  should  provide 
mechanisms  for  the  appraisal  team  to  control  the  interactions. 

•  emphasis  on  the  verification-based  approach  -  Early  identification  and  provision  of 
documented  objective  evidence  by  the  appraised  organization  is  emphasized  to  reduce 
the  extent  of  data  that  must  be  obtained  through  discovery  techniques  by  the  appraisal 
team  during  the  Conduct  Appraisal  phase.  It  is  recommended  that  identification  of  this 
data  by  the  organizational  unit  be  facilitated  to  ensure  that  an  appropriate  and  useful  set 
of  objective  evidence  is  available;  too  much  data  that  is  not  useful  is  just  as  great  a 
problem  as  too  little  data. 

•  reduced  crafting  of  findings  -  In  an  indicator-based  appraisal,  greater  emphasis  is  placed 
on  verification  of  PIIs  and  there  is  less  need  overall  for  crafting  findings.  The  appraisal 
team  need  not  spend  time  generating  findings  that  simply  acknowledge  satisfactory 
implementations  or  the  existence  of  artifacts,  but  can  focus  more  on  identifying 
weaknesses  or  significant  strengths. 

•  consensus  -  At  the  discretion  of  the  appraisal  team  leader,  mini-teams  are  given  the 
authority  to  reach  consensus  on  practice  implementation  at  the  instantiation  level;  full 
team  consensus  is  required  for  aggregation  to  the  organizational  unit  level.  (See  Figure  I- 
2.)  The  characterization  of  practice  implementation  (i.e.,  FI,  LI,  PI,  NI,  NY  found  in 
“Data  Collection,  Rating,  and  Reporting”  on  page  1-25)  can  also  help  facilitate  consensus 
on  whether  implementations  satisfy  model  intent,  either  at  the  instantiation  or 
organizational  unit  level.  The  consensus,  discussion,  and  decision-making  processes  used 
by  the  appraisal  team  can  be  significant  sources  of  inefficiency  if  not  monitored  closely. 

•  effective  data  management  -  The  SCAMPI  A  method  provides  ways  to  collect,  organize, 
and  manage  appraisal  data  efficiently,  and  to  facilitate  the  team  decisions  that  must  be 
made  based  on  the  set  of  objective  evidence.  The  focused  investigation  techniques 
described  earlier  in  this  section  can  help  keep  the  team  oriented  on  what  objective 
evidence  has  been  collected,  what  remains  to  be  collected,  and  how  it  will  be  collected. 
This  process  can  be  greatly  enhanced  through  the  use  of  automated  support  tools.  A 
thorough  understanding  of  progress  toward  sufficiency  of  coverage  can  help  focus  data 
collection.  Interviews,  in  particular,  can  be  shortened  by  focusing  on  specific  data 
collection  needs. 


Several  additional  effective  team  practices  are  targeted  toward  specific  subsets  of  the 
appraisal,  and  are  included  as  suggested  implementation  or  tailoring  options  within  individual 
process  descriptions  in  Part  II. 
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Method  Description 

This  section  provides  an  overview  of  the  SCAMPI  A  method  architecture,  including  appraisal 
phases,  processes,  and  activities.  These  descriptions  are  high-level  abstractions  of  the  process 
descriptions  contained  in  Part  II  of  this  document. 


A  summary  of  the  SCAMPI  A  method  processes  and  activities  for  each  of  the  three  appraisal 
phases  is  contained  in  Tables  T9  through  I- 1 1 . 


The  interactions  among  the  processes  are  depicted  in  the  process  flow  diagrams  in  Figures  1-3 
through  1-5.  These  diagrams  show  the  work  products  that  are  inputs  and  outputs  at  the 
process  level  for  accomplishing  the  purpose  of  the  appraisal.  Additional  analysis  was  done  to 
ensure  that  the  activities  within  each  process  use  and  provide  the  inputs  and  outputs  of  the 
process.  However,  that  detailed  analysis  is  not  presented  here. 


The  process  flows  generally  show  summarized  and  completed  products.  For  instance,  the 
appraisal  input  generated  by  the  Analyze  Requirements  process  initially  is  provided  to  the 
Develop  Appraisal  Plan  process  with  some  elements  missing  that  are  generated  in  other 
processes.  These  inputs  flow  back  to  the  Analyze  Requirements  process  in  the  appraisal  plan. 
The  final  appraisal  input,  as  coordinated  with  the  sponsor,  is  then  produced  as  a  completed 
product.  Additional  administrative  and  support  products,  such  as  appraisal  checklists,  will  be 
produced  but  are  not  included  in  these  diagrams. 


CMU/SEI-2006-HB-002 


1-31 


Table  1-9:  SCAMPI  A  Phase  Summary:  Plan  and  Prepare  for  Appraisal 


Phase 

Process 

Purpose 

Activities 

1  Plan  and 

1 . 1  Analyze  Requirements 

Understand  the  business  needs  of  the  organizational  unit  for  which 

1.1.1  Determine  Appraisal  Obj  ectives 

Prepare  for 
Appraisal 

the  appraisal  is  being  requested.  The  appraisal  team  leader  will 
collect  information  and  help  the  appraisal  sponsor  match  appraisal 
objectives  with  their  business  objectives. 

1.1.2  Determine  Appraisal  Constraints 

1.1.3  Determine  Appraisal  Scope 

1.1.4  Determine  Outputs 

1.1.5  Obtain  C ommitment  to  Appraisal 

Input 

1 .2  Develop  Appraisal  Plan 

Document  requirements,  agreements,  estimates,  risks,  method 
tailoring,  and  practice  considerations  (e.g.,  schedules,  logistics,  and 
contextual  information  about  the  organization)  associated  with  the 
appraisal.  Obtain,  record,  and  make  visible  the  sponsor’s  approval 
of  the  appraisal  plan. 

1.2.1  Tailor  Method 

1 .2.2  Identify  Needed  Resources 

1 .2.3  Determine  Cost  and  Schedule 

1 .2.4  Plan  and  Manage  Logistics 

1 .2.5  Document  and  Manage  Risks 

1 .2.6  Obtain  Commitment  to  Appraisal  Plan 

1 .3  Select  and  Prepare  Team 

Ensure  that  an  experienced,  trained,  appropriately  qualified  team  is 
available  and  prepared  to  execute  the  appraisal  process. 

1.3.1  Identify  Appraisal  Team  Leader 

1.3.2  Select  Team  Members 

1.3.3  Prepare  T earn 

1 .4  Obtain  and  Inventory  Initial 

Obtain  information  that  facilitates  site-specific  preparation.  Obtain 

1.4.1  Obtain  Initial  Objective  Evidence 

Objective  Evidence 

data  on  model  practices  used.  Identify  potential  issue  areas,  gaps,  or 
risks  to  aid  in  refining  the  plan.  Get  preliminary  understanding  of 
the  organizational  unit’s  operations  and  processes. 

1.4.2  Inventory  Objective  Evidence 

1 .5  Prepare  for  Appraisal  Conduct 

Plan  and  document  specific  data-collection  strategies  including 
sources  of  data,  tools  and  technologies  to  be  used,  and  contingencies 
to  manage  risk  of  insufficient  data. 

1.5.1  Perfonn  Readiness  Review 

1.5.2  Prepare  Data  Collection  Plan 

1.5.3  Replan  Data  Collection  (if  needed) 
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Table  1-10:  SCAMPI  A  Phase  Summary:  Conduct  Appraisal 


Phase 

Process 

Purpose 

Activities 

2  Conduct 
Appraisal 

2.1  Prepare  Participants 

Ensure  that  organizational  unit  appraisal  participants 
understand  the  purpose  of  the  appraisal  and  are  prepared 
to  participate. 

2.1.1  Conduct  Participant  Briefing 

2.2  Examine  Objective  Evidence 

Collect  information  about  the  practices  implemented  in 
the  organizational  unit  and  relate  the  resultant  data  to  the 
appraisal  reference  model.  Perform  the  activity  in 
accordance  with  the  data  collection  plan.  Take  corrective 
actions  and  revise  the  data  collection  plan  as  needed. 

2.2.1  Examine  Objective  Evidence  from  Documents 

2.2.2.  Examine  Objective  Evidence  from  Interviews 

2.3  Document  Objective  Evidence 

Create  lasting  records  of  the  infonnation  gathered  by 
identifying  and  then  consolidating  notes,  transfonning 
the  data  into  records  that  document  practice 
implementation,  as  well  as  strengths  and  weaknesses. 

2.3.1  Take/Review/Tag  Notes 

2.3.2  Record  Presence/Absence  of  Objective  Evidence 

2.3.3  Document  Practice  Implementation 

2.3.4  Review  and  Update  the  Data  Collection  Plan 

2.4  Verify  Objective 

Evidence 

Verify  the  implementation  of  the  organizational  unit’s 
practices  for  each  instantiation.  Each  implementation  of 
each  practice  is  verified  so  it  may  be  compared  to 
appraisal  reference  model  practices,  and  the  team 
characterizes  the  extent  to  which  the  practices  in  the 
model  are  implemented. 

2.4.1  Verify  Objective  Evidence 

2.4.2  Characterize  Implementation  of  Model  Practices 

2.5  Validate  Preliminary  Findings 

Validate  the  preliminary  findings,  describing  weaknesses 
(i.e.,  gaps  in  the  implementation  of  model  practices) 
verified.  Weaknesses  in  practice  implementation  are 
validated  with  members  of  the  organizational  unit. 
Exemplary  implementations  of  model  practices  may  be 
highlighted  as  strengths  to  be  included  in  appraisal 
outputs. 

2.5.1  Validate  Preliminary  Findings 

CMU/SEI-2006-HB-002 


1-33 


2.6  Generate  Appraisal  Results 

Rate  goal  satisfaction  based  on  the  extent  of  practice 
implementation  throughout  the  organizational  unit.  The 
extent  of  practice  implementation  is  determined/judged 
based  on  validated  data  (e.g.,  the  three  types  of  objective 
evidence)  collected  from  the  entire  representative  sample 
of  the  organizational  unit.  The  rating  of  capability  levels 
and/or  maturity  levels  is  driven  algorithmically  by  the 
goal  satisfaction  ratings. 

2.6.1  Derive  Findings  and  Rate  Goals 

2.6.2a  Detennine  Process  Area  Capability  Level 

2.6.2b  Detennine  Satisfaction  of  Process  Areas 

2.6.3a  Detennine  Capability  Profile 

2.6.3b  Detennine  Maturity  Level 

2.6.4  Document  Appraisal  Results 

Table  1-11:  SCAMPI  A  Phase  Summary:  Report  Results 


Phase 

Process 

Purpose 

Activities 

3  Report  Results 

3.1  Deliver  Appraisal  Results 

Provide  credible  appraisal  results  that  can  be  used  to 
guide  actions.  Represent  the  strengths  and 
weaknesses  of  the  processes  in  use  at  the  time. 

Provide  ratings  (if  planned  for)  that  accurately  reflect 
the  capability  level/maturity  level  of  the  processes  in 
use. 

3.1.1  Deliver  F inal  F hidings 

3.1.2  Conduct  Executive  Session(s) 

3.1.3  Plan  for  Next  Steps 

3.2  Package  and  Archive  Appraisal 
Assets 

Preserve  important  data  and  records  from  the 
appraisal,  and  dispose  of  sensitive  materials  in  an 
appropriate  manner. 

3.2.1  Collect  Lessons  Learned 

3.2.2  Generate  Appraisal  Record 

3.2.3  Provide  Appraisal  Feedback  to  CMMI  Steward 

3.2.4  Archive  and/or  Dispose  of  Key  Artifacts 
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Figure  1-3:  Process  Flows,  Plan  and  Prepare  Processes 
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Figure  1-4:  Process  Flows,  Conduct  Appraisal  Processes 
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Metrics  Forms 


Baseline  Appraisal  Records, 
Appraisal  Data  Report, 

Method  Evaluation  Report, 

Metrics  Report, 

Field  Observation  Report  (as  necessary), 
Appraisal  Packaging  Checklist 


Figure  1-5:  Process  Flows,  Report  Results  Processes 
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1  Plan  and  Prepare  for  Appraisal 


1.1  Analyze  Requirements 

Purpose  Understand  the  business  needs  of  the  organization  for  which  the  appraisal 

is  being  requested.  The  appraisal  team  leader  will  collect  information  and 
help  the  appraisal  sponsor  match  appraisal  objectives  with  their  business 
objectives. 

Entry  Criteria  •  An  appraisal  sponsor  has  decided  that  a  SCAMPI  appraisal  should  be 

performed. 

•  People  who  can  provide  statements  of  requirements  for  the  appraisal 
are  available. 

Inputs  •  sponsor 

•  initial  requirements  and  constraints 

•  process-related  legacy  information 

Activities  1.1.1  Determine  Appraisal  Objectives 

1.1.2  Determine  Appraisal  Constraints 

1.1.3  Determine  Appraisal  Scope 

1.1.4  Determine  Outputs 

1.1.5  Obtain  Commitment  to  Appraisal  Input 

Outputs  The  appraisal  input 

Outcome  The  decision  to  proceed  with  the  appraisal  based  on  a  shared  understanding 

of  the  appraisal  objectives,  constraints,  outputs,  and  scope. 

Exit  Criteria  •  Initial  contact  between  the  appraisal  sponsor  and  the  appraisal  team 

leader  (i.e.,  an  authorized  or  candidate  SCAMPI  Lead  Appraiser)  has 
occurred. 

•  The  appraisal  team  leader  has  been  given  access  to  members  of  the 
sponsoring  organization. 

•  The  appraisal  input  has  been  approved  by  the  appraisal  sponsor  and 
placed  under  change  management. 

Continued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Key  Points  At  this  early  stage  in  the  process,  gathering  information  that  supports  good 

planning  is  most  important.  Often,  the  appraisal  team  leader  must  educate 
members  of  the  sponsor’s  organization  in  the  purpose  and  role  of 
appraisals. 

Collaborative  consultation  between  the  appraisal  team  leader  and  the 
appraisal  sponsor  is  important  in  this  activity.  The  appraisal  team  leader 
may  be  able  to  simply  interview  the  sponsor  to  get  the  needed  information 
and  reach  agreements.  In  some  settings,  a  series  of  meetings  with  different 
stakeholders  may  be  needed  to  elicit  and  build  consensus  on  the  business 
needs  that  can  be  met  through  a  SCAMPI  A  appraisal. 

Understanding  the  history  of  appraisals  in  the  organization,  especially  the 
organizational  and  appraisal  reference  model  scope  of  past  appraisals,  is 
important  for  understanding  the  requirements  for  the  appraisal  under 
consideration.  The  choices  sponsors  make  about  appraisal  scope  are  often 
tied  to  their  (sometimes  unstated)  priorities  for  process  improvement. 

Metrics  A  number  of  metrics  support  the  appraisal  team  leader’s  monitoring  of  this 

work: 

•  calendar  time  between  initial  contact  and  finalization  of  requirements 

•  effort  expended  to  gather  and  analyze  requirements 

•  number  of  meetings  with  representatives  of  the  sponsoring  and/or 
appraised  organization 

Verification  and  The  exit  criterion  for  this  activity  is  the  formal  approval  of  the  appraisal 
Validation  input  and  its  placement  under  change  management. 

Review  of  the  documented  agreements  resulting  from  the  work  of  this  set 
of  activities  will  serve  to  validate  the  requirements,  which  feed  into 
appraisal  planning. 

Records  The  appraisal  input 

Tailoring  The  experience  of  the  sponsor  with  process  appraisals  will  drive  tailoring 

choices  for  this  process: 

•  A  relatively  inexperienced  appraisal  sponsor  will  need  a  great  deal  of 
information  and  collaborative  consultation  to  provide  meaningful  and 
complete  requirements  for  the  appraisal. 

•  Sponsors  may  have  overly  aggressive  requirements. 

Continued  on  next  page 
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1.1  Analyze  Requirements  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


This  process  is  a  foundation  for  the  success  or  failure  of  the  entire 
appraisal;  it  is  at  this  point  in  the  appraisal  that  the  most  leverage  exists  for 
avoiding  problems  and  issues  downstream.  Gathering  and  understanding 
the  requirements  for  the  conduct  of  a  SCAMPI  A  appraisal  is  vital  to 
making  appropriate  decisions  and  providing  value  to  the  sponsor.  Many 
examples  of  problems  encountered  during  appraisals  can  be  traced  to 
shortcomings  in  the  conduct  of  this  process.  The  extent  to  which  the 
activities  described  here  are  distinct  from  the  activities  described  in  the 
next  process,  Develop  Appraisal  Plan,  will  depend  on  the  strategy  and 
preferences  of  both  the  appraisal  team  leader  and  the  appraisal  sponsor. 


The  objectives  that  motivate  the  conduct  of  an  appraisal  must  be  well 
understood  so  that  appropriate  participants,  tailoring  decisions,  and 
appraisal  outputs  can  be  selected.  The  constraints  that  shape  the  appraisal 
enactment,  in  light  of  the  objectives,  may  limit  achievement  of  the  desired 
result  if  they  are  not  adequately  understood  and  negotiated.  A  clear 
agreement  regarding  appraisal  outputs  and  their  intended  usage  will  help 
maintain  the  sponsorship  needed  for  conducting  the  appraisal  and  acting  on 
the  results.  Establishing  agreement  on  these  objectives,  constraints, 
outputs,  and  intended  usage  forms  the  basis  for  a  commitment  to  the  plan 
for  conducting  the  appraisal. 


♦ 
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1.1.1  Determine  Appraisal  Objectives 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  business  needs  for  process  improvement  drive  the  requirements  for  the 
conduct  of  any  given  appraisal  and  generally  include  one  or  more  of  three 
closely  related  factors: 

•  reducing  costs 

•  improving  quality 

•  decreasing  time  to  market 

The  fundamental  premise  of  process  improvement  is  that  organizational 
processes  significantly  impact  these  factors. 

Obtaining  a  fair  and  objective  characterization  of  the  processes  in  use  in 
the  organization(s)  is  the  essential  reason  for  conducting  an  appraisal.  In 
addition  to  this  motivation,  a  sponsor’s  desire  to  conduct  an  appraisal 
could  be  driven  by  one  or  more  of  the  following  business-related 
objectives: 

•  Document  a  credible  benchmark  that  reflects  successful  process 
improvement. 

•  Evaluate  areas  of  potential  risk  that  may  affect  the  performance  of  the 
organization. 

•  Involve  members  of  the  appraised  organization  in  improving  the 
performance  of  the  process. 

•  Support  specific  decisions  related  to  the  direction  of  a  new  or  existing 
improvement  program. 

•  Motivate  a  supplier  to  focus  on  process  issues  that  affect  contract 
performance. 


The  appraisal  team  leader  or  designee  shall 

•  identify  sponsor  and  relevant  stakeholders,  and  establish 
communication 

•  document  the  business  objectives  provided  by  the  sponsor  and  the 
specific  appraisal  objectives 

•  ensure  the  alignment  of  the  appraisal  objectives  to  the  business 
objectives 

•  determine  and  document  the  appraisal  usage  mode  (i.e.,  Internal 
Process  Improvement,  Supplier  Selection,  or  Process  Monitoring) 


At  least  one  communication  between  the  appraisal  team  leader  and  sponsor 
is  required. 


Con  tin  ued  on  next  page 
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1.1.1  Determine  Appraisal  Objectives  (continued) 


Optional  None 

Practices 


Implementation  Organizations  with  experience  in  the  use  of  appraisals  may  have  a  clear  set 
Guidance  of  appraisal  objectives  identified  in  advance  of  contacting  an  appraisal 

team  leader. 

In  some  cases,  the  usage  mode  will  be  self-evident;  however,  there  may  be 
instances  in  which  the  appraisal  sponsor  either  may  not  be  sure  or  may 
have  made  an  assumption  that  is  not  founded  on  fact.  The  appraisal  team 
leader  is  responsible  for  ensuring  that  the  best  choice  of  usage  mode  is 
made  consistent  with  the  sponsor’s  input  and  direction. 

Depending  on  the  structure  of  the  appraised  organization,  as  well  as  the 
usage  mode,  it  is  often  important  to  distinguish  the  role  of  senior  site 
manager  from  that  of  the  appraisal  sponsor.  For  some  appraisals,  these  two 
roles  are  encompassed  in  the  duties  of  a  single  person.  For  other  appraisals, 
these  two  roles  may  represent  two  people  working  many  time  zones  away 
from  each  other. 


♦ 
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1.1.2  Determine  Appraisal  Constraints 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 

Optional 

Practices 


The  constraints  within  which  the  appraisal  must  be  conducted  are 
determined  based  on  a  dialog  between  the  appraisal  team  leader  and  the 
appraisal  sponsor  and/or  senior  site  manager.  This  dialog  typically  is  an 
iterative  process  in  which  the  preferences  of  the  appraisal  sponsor,  the 
limits  of  the  method,  and  the  consequent  resource  requirements  are 
balanced  against  each  other  to  arrive  at  an  optimal  set  of  appraisal  input 
parameters. 


The  appraisal  team  leader  or  designee  shall 

•  establish  high-level  cost  and  schedule  constraints 

•  determine  which  process  areas  (PAs)  and  organizational  entities  are  to 
be  included 

•  determine  minimum,  maximum,  or  specific  sample  size  or  coverage 
that  is  desired  for  the  appraisal 

•  negotiate  constraints  and  objectives  with  stakeholders  to  ensure 
feasibility 

•  document  negotiated  constraints  to  be  met 


Constraints  identified  by  the  appraisal  input  must  be  negotiated  between 
the  sponsor  and  the  appraisal  team  leader. 


Document  the  rationale  for  choices  made  and  the  associated  tradeoffs  as  a 
resource  for  later  planning  and  future  appraisals.  If  the  organization  is 
aware  of  any  potential  alternative  practices,  it  is  recommended  that  these 
practices  be  brought  to  the  attention  of  the  appraisal  team  leader  during 
this  activity.  See  Appendix  C,  Alternative  Practice  Identification  and 
Characterization  Guidance,  for  information  on  identifying  acceptable 
alternative  practices. 


Continued  on  next  page 
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1.1.2  Determine  Appraisal  Constraints  (continued) 


Implementation 

Guidance 


Page  11-8 


Constraints  on  cost  and  schedule  identified  during  this  early  stage  of  the 
appraisal  are  expected  to  be  high  level  and  not  detailed  estimates.  They 
may  take  the  form  of  statements  such  as  “We  need  this  done  in  Q4,”  “You 
can’t  use  more  than  five  of  my  people  on  the  team,”  and  “I  can’t  afford  to 
have  it  last  more  than  a  month.” 

Practical  limitations  relating  to  time,  cost,  and  effort  are  clarified  and 
negotiated  in  the  context  of  other  requirements  the  sponsor  has.  The 
business  context  in  which  the  appraisal  is  conducted  drives  choices  that  the 
appraisal  team  leader  must  make.  For  example,  if  virtual  methods  (e.g., 
video  conferences,  teleconferences,  and  other  similar  technology)  are  to  be 
used  to  conduct  appraisal  activities,  the  constraints  imposed  by  these 
methods  should  be  discussed,  documented,  and  taken  into  account  as  the 
appraisal  is  planned.  Appraisals  should  not  be  conducted  in  isolation  from 
other  activities  relating  to  process  management  and  improvement.  The 
needs  of  relevant  stakeholders,  be  they  acquisition  organizations  or 
division  heads  managing  an  engineering-related  process  group,  often  place 
requirements  on  the  conduct  of  the  appraisal. 
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1.1.3  Determine  Appraisal  Scope 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  appraisal  scope  consists  of  the  appraisal  reference  model  scope  and  the 
organizational  scope  to  be  examined  during  the  appraisal.  The  reference 
model  scope  must  be  determined  and  documented  early  in  the  planning 
process,  using  the  staged  representation  or  the  continuous  representation. 
In  conjunction  with  the  appraisal  sponsor,  the  appraisal  team  leader  is 
responsible  for  making  decisions  regarding  the  PAs  included  in  the  scope 
of  the  appraisal  and  the  model  representation.  The  selection  of  appraisal 
outputs  should  be  driven  by  the  understanding  of  their  intended  use, 
established  during  the  requirements  analysis  activity,  and  may  dictate  some 
selections  in  reference  model  scope. 

The  organizational  unit  is  that  part  of  an  organization  that  is  the  subject  of 
an  appraisal  and  to  which  the  appraisal  results  will  be  generalized.  This 
organizational  unit  may  include  the  entire  organization,  one  or  more 
divisions  within  the  organization,  or  one  or  more  projects  within  the 
organization.  The  organizational  scope  consists  of  the  projects  and  support 
functions  sampled  to  provide  examples  (or  instantiations)  of  practices  used 
in  appropriate  contexts  within  the  boundaries  of  the  organizational  unit. 


The  appraisal  team  leader,  in  conjunction  with  the  appraisal  sponsor  and/or 
the  sponsor’s  designee,  shall 

•  determine  and  document  the  reference  model  scope  and  representation 
to  be  used  for  the  appraisal 

•  determine  and  document  the  organizational  unit  to  be  investigated 
during  the  appraisal 

•  determine  and  document  the  organizational  scope  of  the  appraisal 

•  identify  and  document  the  names  of  individuals  who  will  participate 
in  the  appraisal 


The  appraisal  reference  model  scope  shall  include  the  PAs  and  associated 
maximum  capability  level  (i.e.,  the  generic  goals  that  will  be  rated  for  each 
PA  within  the  scope  of  the  appraisal)  and/or  maturity  level  that  will  be 
investigated  by  the  appraisal  team. 

The  reference  model  scope  of  the  appraisal  shall  encompass  at  least  one 
PA.  All  generic  goals  and  specific  goals  up  to  and  including  the  target 
capability  level  and/or  maturity  level  for  each  selected  PA  must  be  rated; 
individual  goals  within  a  PA  cannot  be  excluded. 

When  a  PA  is  determined  to  be  outside  of  the  organizational  unit's  scope  of 
work,  the  PA  is  designated  as  "not  applicable."  Any  PA  designated  as  "not 
applicable"  and  the  rationale  for  its  exclusion  must  be  documented  in  the 
appraisal  input  and  Appraisal  Disclosure  Statement  (ADS). 


Con  tin  ued  on  next  page 
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1.1.3  Determine  Appraisal  Scope  (continued) 


Parameters 
and  Limits 
(continued) 


The  organizational  scope  of  the  appraisal  must  include  sample  projects  and 
support  groups  that  are  representative  of  the  implemented  processes  and 
functional  areas  being  investigated  within  the  organizational  unit  and  that 
operate  within  a  coherent  process  context.  The  rationale  for  selecting  these 
sample  projects  and  support  groups  as  representative  of  the  organizational 
unit  must  be  documented. 

Organizational  unit  size  (i.e.,  number  of  people  and  number  of  projects) 
and  sizes  of  projects  and  support  groups  (i.e.,  number  of  people)  in  the 
organizational  scope  must  be  documented  as  well  as  the  percentage  ratio  of 
these  two  measures: 

•  Population  %:  the  number  of  people  in  the  organizational  scope 
divided  by  the  number  of  people  in  the  organizational  unit  (xlOO) 

•  Project  %:  the  number  of  projects  in  the  organizational  scope  divided 
by  the  number  of  projects  in  the  organizational  unit  (xlOO) 

Critical  factors  that  influence  implementation  of  practices  in  projects  and 
functions  within  the  organizational  unit  must  also  be  understood  and 
identified.  (Possible  critical  factors  are  discussed  in  the  Implementation 
Guidance  below.) 

Sample  projects  and  support  groups  selected  to  form  the  organizational 
scope  (i.e.,  the  combination  of  focus  and  non-focus  projects  and  support 
functions)  must  represent  all  critical  factors  identified  for  the 
organizational  unit  to  which  the  results  will  be  attributed.  The  coverage  of 
the  organizational  critical  factors  provided  by  these  sample  projects  and 
support  groups  in  the  organizational  scope  in  relation  to  the  organizational 
unit  must  be  documented,  in  quantitative  terms,  in  the  appraisal  input  and 
ADS. 

Each  sample  project  or  support  group  in  the  planned  organizational  scope 
of  the  appraisal  must  be  one  of  the  three  types  listed  below: 

•  Focus  projects  must  provide  objective  evidence  for  every  PA  within 
the  model  scope  of  the  appraisal  which  addresses  model  practices 
applicable  to  those  projects. 

•  Non-focus  projects  must  provide  objective  evidence  for  one  or  more 
PAs  within  the  model  scope  of  the  appraisal  which  address  practices 
performed  on  projects. 

Support  functions  must  provide  objective  evidence  for  practices  within  the 
model  scope  of  the  appraisal  that  address  organizational  infrastructure  or 
functions. 


Con  tin  ued  on  next  page 
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1.1.3  Determine  Appraisal  Scope  (continued) 


Parameters 
and  Limits 
(continued) 


Optional 

Practices 


In  appraisals  where  the  reference  model  scope  includes  any  project-related 
PA,  the  organizational  scope  must  include  at  least  one  focus  project.  If  the 
organizational  unit  includes  more  than  3  projects,  then  the  organizational 
scope  must  include  sufficient  focus  projects  and  non-focus  projects  to 
generate  at  least  3  instances  of  each  practice  in  each  project-related  PA  in 
the  model  scope  of  the  appraisal. 

Projects,  categories,  or  groups/functions  that  are  specifically  excluded 
from  the  appraisal  must  be  identified  in  the  appraisal  input  and  in  the  ADS 
as  well  as  the  justification  for  their  exclusion.  This  identification  includes 
legacy  projects  not  using  current  organizational  processes  and  projects 
waived  from  using  current  organizational  processes. 

As  needed,  the  appraisal  team  may  seek  clarification  or  data  from  other 
projects  or  support  functions  within  the  organizational  unit.  These  projects 
or  support  functions  must  also  be  identified  in  the  ADS. 

The  representative  samples  to  be  investigated  during  the  appraisal  will 
drive  the  selection  of  participants  needed  to  provide  sources  of  objective 
evidence.  An  initial  determination  of  appraisal  participants,  by  name  and 
role,  must  be  negotiated  with  the  appraisal  sponsor  and/or  the  senior  site 
manager  as  part  of  the  early  determination  of  organizational  scope.  This 
initial  determination  will  be  refined  later  during  detailed  appraisal 
planning. 

If  the  Conduct  Appraisal  phase  is  to  be  performed  using  incremental 
subsets  of  the  organizational  unit  or  the  model,  the  appraisal  plan  must 
identify  the  organizational  scope  and  appraisal  reference  model  scope  for 
each  increment. 

Delta  appraisals  are  not  permitted.  A  delta  appraisal  is  defined  as  a  second 
appraisal  performed  on  a  subset  of  an  original  appraisal  model  scope  after 
correcting  weaknesses  reported  in  the  previous  appraisal,  and  then 
combining  the  results  of  the  second  appraisal  with  the  results  of  the  parts 
of  the  first  appraisal  that  were  not  investigated  in  the  second  appraisal  to 
get  new  results. 


Use  broad-based  survey  instruments  or  a  PII  database  to  characterize  the 
population  of  projects  or  divisions  in  an  organization  to  aid  in  determining 
the  organizational  scope  of  the  appraisal. 


Con  tin  ued  on  next  page 
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1.1.3  Determine  Appraisal  Scope  (continued) 


Implementation 

Guidance 


The  selection  of  the  appraisal  reference  model  representation  should  have 
been  discussed  during  the  setting  of  appraisal  objectives  because  the 
representation  selected  may  impact  the  achievability  of  these  objectives. 

Clearly,  a  broadly-defined  organizational  unit  (e.g.,  a  multi-national 
enterprise)  will  require  collecting  and  analyzing  significantly  more 
objective  evidence  than  a  narrowly  defined  organizational  unit  (e.g.,  a 
specific  product  line  within  a  specific  business  unit  at  a  single  geographical 
location). 

The  organizational  unit  to  which  appraisal  results  will  be  attributed  should 
be  described  accurately  in  all  statements  made  by  the  appraisal  team  leader 
and  sponsor.  It  is  the  responsibility  of  the  appraisal  team  leader  to 
understand  the  larger  organizational  context  in  which  the  appraised 
organizational  unit  resides.  Familiarity  with  the  nature  of  departmental 
structures,  matrixed  subject  matter  expert  groups,  integrated  product  teams, 
program  and  project  groupings,  or  product  line  implications  that  may  affect 
the  interpretation  of  appraisal  outcomes  will  aid  in  obtaining  this 
understanding.  The  organizational  unit  should  be  documented  in  the 
clearest  terms  possible,  given  the  nature  of  the  organizational  structure  in 
place.  It  is  often  difficult  to  specify  unambiguous  boundaries  without 
resorting  to  naming  individual  people  in  some  organizations.  Information 
about  the  organizational  unit  should  be  documented  in  a  way  that  allows 
future  appraisal  sponsors  to  replicate  (to  the  extent  possible)  the 
organizational  unit  appraised.  This  information  should  be  in  the  appraisal 
plan,  and  used  (in  summary  form  if  needed)  in  briefing  the  appraisal  team 
and  appraisal  participants. 

Other  examples  of  critical  factors  include: 

•  application  domains  (or  lines  of  business) 

•  geographical  breadth 

•  disciplines  (e.g.,  systems  engineering,  software  engineering,  or 
hardware  engineering) 

•  effort  types  (e.g.,  development,  maintenance,  or  services) 

•  project  types  (e.g.,  legacy  or  new  development) 

•  customer  types  (e.g.,  commercial  or  government  agency) 

•  lifecycle  models  in  use  within  the  organization  (e.g.,  spiral, 
evolutionary,  waterfall,  or  incremental) 


Continued  on  next  page 
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1.1.3  Determine  Appraisal  Scope  (continued) 


Implementation 

Guidance 

(continued) 


The  following  are  examples  of  documenting  the  quantitative  coverage  of 
size  and  critical  factors  provided  by  the  sample  projects  or  support  groups 
in  the  organizational  scope  relative  to  the  organizational  unit: 

•  Projects  within  the  organizational  scope  include  50%  of  the 
employees  in  the  organizational  unit. 

•  Projects  within  the  organizational  scope  represent  60%  of  the  lines  of 
business  within  the  organizational  unit. 

•  Projects  within  the  organizational  scope  represent  50%  of  the 
geographic  sites  included  in  the  organizational  unit. 

The  appraisal  team  leader  should  work  with  representatives  from  the 
organization  to  document  a  clear  statement  of  the  reference  model  and 
organizational  scope  of  the  appraisal.  The  appraisal  reference  model  scope 
should  be  documented  using  a  list  of  PAs  to  be  included  in  the  appraisal  as 
well  as  the  model  components  to  be  rated  by  the  appraisal  team. 

If  additional  projects  or  support  groups  within  the  organizational  unit  (e.g., 
other  projects  or  instances  of  support  functions)  provide  evidence  during 
the  conduct  of  the  appraisal,  the  organizational  scope  identified  in  the  plan 
and  the  organizational  scope  documented  in  the  ADS  will  be  different.  The 
organizational  scope  documented  in  the  ADS  will  include  the  projects  and 
support  groups  that  provided  the  additional  evidence. 
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1.1.4  Determine  Outputs 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Identify  the  specific  appraisal  outputs  to  be  produced.  Some  appraisal 
outputs  are  required  and  additional  outputs  are  tailorable  (see  Parameters 
and  Limits  and  Optional  Practices). 

Obtain  information  to  answer  the  following  questions: 

•  What  ratings  will  be  generated  during  the  appraisal? 

•  Will  a  final  report  be  written  to  document  appraisal  results? 

•  Will  recommendations  on  how  to  address  specific  findings  be 
generated  and  reported? 


The  appraisal  team  leader  or  designee  shall 

•  review  required  SCAMPI  A  outputs  with  the  appraisal  sponsor 

•  review  and  select  optional  SCAMPI  A  outputs  with  the  appraisal 
sponsor 

•  determine  the  recipients  of  appraisal  outputs  based  on  sponsor 
instructions 


Required  SCAMPI  A  outputs  include 

•  appraisal  record  (see  activity  3.2.2,  Generate  Appraisal  Record) 

•  ADS  (see  activity  2.6.4,  Document  Appraisal  Results) 

•  CMMI  Steward  data  (see  activity  3.2.3,  Provide  Appraisal  Feedback 
to  CMMI  Steward) 

As  stated  in  the  ARC,  at  least  all  the  goals  for  the  PA  or  PAs  within  the 
model  scope  must  be  rated  for  the  organizational  unit,  although  the  choice 
may  be  made  to  not  disclose  the  ratings  to  anyone  other  than  the  appraisal 
sponsor.  Ratings  for  individual  disciplines  or  for  individual  projects,  unless 
the  project  is  the  organizational  unit,  are  not  allowed. 

The  sponsor  shall  receive  the  appraisal  record,  which  includes 

•  final  findings,  including  statements  of  strengths  and  weaknesses 
documented  for  every  PA  investigated 

•  all  ratings  planned  for  and  generated  by  the  team 

•  the  ADS 

The  appraisal  team  leader  and  sponsor  are  required  to  sign  the  ADS. 

Decisions  reached  on  appraisal  outputs,  including  what  ratings  will  be 
reported,  are  documented  in  the  appraisal  input. 


Continued  on  next  page 
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1.1.4  Determine  Outputs  (continued) 


Optional  The  appraisal  sponsor  may  request  that  additional  rating  outputs  be 

Practices  generated  as  a  result  of  the  appraisal.  Typical  rating  outputs  that  might  be 

selected  include 

•  maturity  level  and/or  capability  level  ratings 

•  PA  Satisfaction/Capability  Level  Profiles 

•  increment-specific  findings 

•  project-level  findings 

•  other  (non-typical)  outputs  desired 

Many  of  these  optional  appraisal  outputs  are  discussed  further  in  process 
2.6,  Generate  Appraisal  Results. 

The  sponsor  may  also  request  that  other  products  be  generated  as  appraisal 
outputs.  Typical  products  that  might  be  requested  include  (see  activity 
3.1.3,  Plan  for  Next  Steps): 

•  appraisal  final  report 

•  recommendations  for  taking  action  on  the  appraisal  results 

•  process  improvement  action  plan 


Implementation  Goal  satisfaction  ratings  for  both  specific  goals  and  generic  goals  of  the 

Guidance  PAs  within  the  scope  of  the  appraisal  are  a  minimum  requirement. 

Capability  and/or  maturity  level  ratings  are  optional. 

While  statements  of  findings  are  a  required  output  of  the  method,  creating 
a  written  report  that  elaborates  on  the  findings  is  optional.  The  sponsor 
should  decide  if  resources  are  to  be  spent  creating  this  artifact.  Similarly, 
the  task  of  creating  recommendations  to  address  issues  uncovered  in  the 
appraisal  may  require  expertise  that  is  not  represented  on  the  appraisal 
team  in  some  cases.  The  characteristics  of  the  appraised  organization  and 
the  constraints  that  shape  its  improvement  program  should  be  carefully 
considered  when  making  process  improvement  recommendations. 
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1.1.5  Obtain  Commitment  to  Appraisal  Input 


Activity 

Description 

Required 

Practices 


Parameters 
and  Limits 


The  appraisal  sponsor  formally  approves  the  appraisal  input,  and  this  set  of 
information  is  placed  under  change  management. 


The  appraisal  team  leader  or  designee  shall 

•  record  required  information  in  the  appraisal  input 

•  obtain  sponsor  approval  of  the  appraisal  input 

•  manage  changes  to  the  appraisal  input,  obtaining  sponsor  approval  of 
changes 


The  appraisal  input  may  be  generated  incrementally  throughout  planning, 
but  must  be  approved  prior  to  the  start  of  data  collection.  At  a  minimum, 
the  appraisal  input  shall  provide  the  information  needed  to  address  the 
following: 

•  the  identity  of  the  appraisal  sponsor  and  the  relationship  of  the 
sponsor  to  the  organizational  unit  being  appraised 

•  the  appraisal  purpose,  including  alignment  with  business  objectives 
(see  activity  1.1.1,  Determine  Appraisal  Objectives) 

•  the  appraisal  scope  (see  activity  1.1.3,  Determine  Appraisal  Scope) 

-  the  organizational  unit  being  appraised 

-  the  organizational  scope  of  the  appraisal 

-  critical  factors  affecting  the  appraisal 

•  the  process  context,  which  includes,  at  a  minimum 

-  organizational  unit  size  and  demographics 

-  application  domain,  size,  criticality,  and  complexity 

-  high-priority  characteristics  (e.g.,  time  to  market,  feature  richness, 
and  reliability)  of  the  products  and  services  of  the  organizational 
unit 

•  appraisal  constraints  (see  activity  1.1.2,  Determine  Appraisal 
Constraints),  which  include,  at  a  minimum 

-  availability  of  key  resources  (e.g.,  staffing,  funding,  tools,  and 
facilities) 

-  schedule  constraints 

-  the  maximum  amount  of  time  to  be  used  for  the  appraisal  (The 
maximum  time  to  perform  the  Conduct  Appraisal  phase  is  90  days.) 

-  specific  PAs  or  organizational  entities  to  be  excluded  from  the 
appraisal 

-  the  maximum,  minimum,  or  specific  sample  size  or  coverage 
desired  for  the  appraisal 

-  ownership  of  appraisal  results  and  any  restrictions  on  their  use 

-  controls  on  information  resulting  from  a  confidentiality  agreement 

-  non- attribution  of  appraisal  outputs  to  individuals 

•  the  identity  of  the  appraisal  reference  models  used  (version, 
discipline,  and  representation) 
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1.1.5  Obtain  Commitment  to  Appraisal  Input  (continued) 


Parameters 
and  Limits 
(continued) 


•  the  identity  and  affdiation  of  the  SCAMPI  Lead  Appraiser  who  is  to 
be  the  appraisal  team  leader  for  the  appraisal 

•  the  identity  and  affiliation  of  the  appraisal  team  members  and  their 
specific  appraisal  responsibilities 

•  the  identity  (i.e.,  name  and  organizational  affiliation)  of  appraisal 
participants  and  support  staff,  and  their  specific  responsibilities  for  the 
appraisal 

•  any  additional  information  to  be  collected  during  the  appraisal  to 
support  the  achievement  of  the  appraisal  objectives 

•  a  description  of  the  planned  appraisal  outputs  (see  activity  1.1.4, 
Determine  Outputs),  including  ratings  to  be  generated 

•  anticipated  follow-on  activities  (e.g.,  reports,  appraisal  action  plans, 
or  re-appraisal) 

•  planned  tailoring  of  SCAMPI  A  and  associated  tradeoffs,  including 
the  sample  size  or  coverage  of  the  organizational  unit 

•  appraisal  usage  mode  (i.e.,  Internal  Process  Improvement,  Supplier 
Selection,  or  Process  Monitoring) 


Optional  None 

Practices 


Implementation  An  appraisal  team  leader’s  ability  to  build  and  maintain  commitment  from 
Guidance  the  sponsor  and  the  members  of  the  sponsoring  organization  is  a  major 

factor  contributing  to  the  success  of  the  appraisal.  The  process  of 
understanding  the  requirements  and  constraints  should  yield  a  series  of 
agreements  that  form  an  input  to  the  appraisal  plan.  Based  on  the  judgment 
of  the  appraisal  team  leader,  these  agreements  may  be  covered  in  a  formal 
(signed)  document  that  forms  a  basis  for  future  activities.  More  typically, 
the  appraisal  team  leader  maintains  a  record  of  interactions  with  the 
sponsor,  which  are  incorporated  into  the  appraisal  plan  as  it  is  drafted. 

The  appraisal  team  leader  and  the  sponsor  should  have  verbal  agreement 
on  the  items  discussed  above,  and  these  items  should  be  documented  in 
some  way.  The  formality  of  the  documentation  may  range  from  simple 
meeting  minutes  maintained  by  the  appraisal  team  leader,  to  a  more  formal 
memorandum  of  understanding  or  other  vehicle  that  documents 
agreements  and  provides  traceability.  It  is  expected  that  the  appraisal  plan 
will  be  used  to  document  important  issues  pertaining  to  requirements. 
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1.2  Develop  Appraisal  Plan 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Document  the  results  of  appraisal  planning  including  the  requirements, 
agreements,  estimates,  risks,  method  tailoring,  and  practical  considerations 
(e.g.,  schedules,  logistics,  and  contextual  information  about  the 
organization)  associated  with  the  appraisal.  Obtain  and  record  the 
sponsor’s  approval  of  the  appraisal  plan. 


The  appraisal  sponsor  and  appraisal  team  leader  have  agreed  to  proceed 
with  appraisal  planning  based  on  a  common  understanding  of  the  key 
parameters  that  drive  the  planning  process. 


Documented  agreements,  reflected  in  the  appraisal  input,  that  support  a 
common  understanding  of  appraisal  objectives  and  key  appraisal-planning 
parameters 


1.2.1  Tailor  Method 

1 .2.2  Identify  Needed  Resources 

1.2.3  Determine  Cost  and  Schedule 

1.2.4  Plan  and  Manage  Logistics 

1 .2.5  Document  and  Manage  Risks 

1 .2.6  Obtain  Commitment  to  Appraisal  Plan 


•  approved  appraisal  plan 

•  strategy  for  managing  logistics 

•  strategy  for  preparing  the  organization(s) 

•  schedule 

•  interview  plan 

•  team  assignments 


The  sponsor  and  appraisal  team  leader  agree  on  technical  and  non¬ 
technical  details  for  the  planned  appraisal.  The  plan  is  refined  in 
conjunction  with  performing  the  other  Planning  and  Preparation  phase 
activities.  This  agreement  is  documented  and  reviewed  by  affected 
stakeholders  as  appropriate. 


The  final  appraisal  plan  is  reviewed  and  approved. 
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1.2  Develop  Appraisal  Plan  (continued) 


Key  Points  Skilled  appraisal  team  leaders  will  effectively  develop  and  use  outputs 

from  the  other  Planning  and  Preparation  phase  activities  to  achieve  clarity 
of  the  shared  vision  necessary  to  make  the  tradeoffs  and  decisions  resulting 
in  a  final  plan.  This  activity  is  an  important  opportunity  for  the  appraisal 
team  leader  to  demonstrate  process  discipline,  as  well  as  the  type  of  careful 
planning  described  in  the  CMMI  appraisal  reference  model.  Experienced 
appraisal  team  leaders  will  leverage  data,  templates,  and  assets  (developed 
through  their  own  experience)  to  improve  the  completeness  and 
effectiveness  of  the  appraisal  plan,  recognizing  the  return  on  investment 
that  will  be  obtained  through  smooth  and  efficient  appraisals. 

Tools  include  an  appraisal  plan  template,  samples,  and  embedded 
procedural  guidance  in  planning  templates.  Estimation  worksheets  and 
methods  for  assessing  the  impact  of  appraisal  constraints  are  also  quite 
useful. 

Metrics  •  calendar  time  spanned  by  the  activity 

•  effort  consumed  in  carrying  out  the  activities  of  this  process 

•  level  and  frequency  of  changes  to  the  appraisal  plan 

•  comparison  of  actual  effort  for  this  activity  with  historical  data 
accumulated  by  the  appraisal  team  leader 

•  review  of  the  appraisal  plan  by  affected  stakeholders 

•  sponsor’s  approval  of  the  plan 

Records  •  estimation  worksheets  (if  used) 

•  appraisal  plan  (see  activity  1.2.6,  Obtain  Commitment  to  Appraisal 
Plan,  for  a  detailed  list  of  plan  contents) 

Tailoring  •  In  some  applications,  planning  templates  and  procedures  in  routine 

use  within  the  organization  can  be  adapted  to  the  needs  of  the 
appraisal.  This  approach  aids  communication  as  well  as  local 
ownership  of  the  process. 

•  A  structured  planning  workshop  may  be  of  benefit  for  organizations 
with  limited  appraisal  experience.  Such  a  workshop  is  a  valuable 
opportunity  to  discover  risks  as  well  as  mitigation  strategies. 

Continued  on  next  page 
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1.2  Develop  Appraisal  Plan  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  appraisal  plan  will  guide  and  define  the  execution  of  the  appraisal 
such  that  it  is  in  concert  with  the  business  needs  and  constraints.  An  initial 
plan  can  be  generated  immediately  following  consultation  with  the 
sponsor.  Further  refinement  is  done  as  detailed  planning  occurs  and  new 
information  comes  to  light  in  executing  appraisal  planning  and  preparation. 
A  final  appraisal  plan  must  be  completed  prior  to  the  completion  of 
process  1.5,  Prepare  for  Appraisal  Conduct.  Typically,  resources,  method 
tailoring,  model-related  decisions,  and  a  planned  list  of  outputs  are 
finalized  early  on,  while  cost,  schedule,  and  logistics  are  finalized  later  in 
the  Plan  and  Prepare  for  Appraisal  phase. 

The  appraisal  input  is  a  necessary  input  to  the  appraisal-planning  process. 
While  it  may  not  be  necessary  to  formally  separate  the  requirements 
analysis  activities  from  the  activities  described  in  this  section,  prior 
understanding  of  the  appraisal  requirements  is  a  necessary  input  to  this 
process.  The  plan  for  the  appraisal  provides  an  important  vehicle  for 

•  documenting  agreements  and  assumptions 

•  establishing  and  maintaining  sponsorship 

•  tracking  and  reporting  the  performance  of  the  appraisal  process 

•  reinforcing  commitments  at  key  points  in  the  appraisal  process 

The  distinction  between  the  appraisal  input  and  appraisal  plan  is  intended 
to  separate  the  key  appraisal  requirements  and  strategic  objectives,  which 
require  high  sponsor  visibility  and  change  control  approval,  from  the 
tactical  planning  details  necessary  to  implement  and  satisfy  these 
objectives.  While  sponsor  visibility  into  the  appraisal  plan  is  necessary, 
revisions  are  typically  low-level  implementation  details  and  do  not 
ordinarily  require  sponsor  re-approval.  In  practical  use,  the  appraisal  input 
is  often  packaged  as  a  component  of  the  appraisal  plan,  and  a  single 
sponsor  signature  can  serve  as  approval  for  both. 


This  process  is  composed  of  six  activities  summarized  here  and  described 
below.  The  scope  of  the  appraisal  is  defined  in  terms  of  (a)  the  portion  of 
the  appraisal  reference  model  that  will  be  investigated  and  (b)  the  bounds 
of  the  organizational  unit  for  which  the  results  can  be  considered  valid 
(e.g.,  a  project,  a  product  line,  an  operating  division,  a  business  unit,  or  an 
entire  global  enterprise).  Method-tailoring  choices  are  made  to  most 
effectively  achieve  appraisal  objectives  within  defined  constraints  of  time, 
effort,  and  cost.  The  resources  required  to  carry  out  the  appraisal  are 
identified.  The  cost  and  schedule  are  negotiated  and  recorded.  The  details 
of  logistics,  particularly  for  the  Conduct  Appraisal  phase,  are  documented. 
Risks  and  risk-mitigation  plans  are  identified  and  documented.  Completion 
of  these  activities  results  in  a  well-defined,  achievable  appraisal  plan. 
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1.2.1  Tailor  Method 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Tailoring  of  SCAMPI  A  includes 

•  selection  of  choices  (if  any)  within  the  Required  Practices 

•  setting  parameters  that  are  allowed  to  vary  within  the  Parameters  and 
Limits 

•  inclusion  of  Optional  Practices 

Because  SCAMPI  A  is  designed  to  apply  to  a  wide  range  of  appraisal 
applications,  the  tailoring  activity  is  one  that  deserves  careful  and 
thoughtful  attention. 

This  document  is  designed  to  clearly  indicate  which  aspects  of  the  method 
are  required  and  which  are  tailorable.  The  Parameters  and  Limits  and 
Optional  Practices  sections  of  each  activity  description  provide  discussions 
of  tailoring  options,  in  context. 

In  addition,  the  appraisal  usage  mode  will  determine  some  tailoring 
choices. 


The  appraisal  team  leader  or  designee  shall 

•  review  and  select  tailoring  options  within  each  activity 

•  ensure  that  the  tailoring  decisions  are  self-consistent  and  that  they  are 
appropriate  in  light  of  the  appraisal  objectives  and  constraints 

•  document  the  tailoring  decisions  made 


The  structure  of  the  MDD  clarifies  which  SCAMPI  A  features  are 
required,  either  as  a  direct  derivative  of  ARC  requirements  or  as  SCAMPI 
A  requirements.  Parameters  and  Limits  sections  define  the  allowable 
variation  within  these  method  requirements.  Tailoring  guidance  and 
Implementation  Guidance  are  provided  to  assist  with  tuning  the  method  to 
fit  sponsor  objectives  and  appraisal  constraints.  Method  tailoring  and 
implementation  options  must  be  selected  and  implemented  in  a  way  that 
does  not  violate  SCAMPI  A  Required  Practices. 


Con  tin  ued  on  next  page 
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1.2.1  Tailor  Method  (continued) 


Optional  Provide  the  sponsor  with  more  than  one  candidate  scenario  for  the 

Practices  appraisal  and  help  them  select  among  the  options. 

Alternatively,  the  appraisal  team  leader  may  define  a  tailored  instance  of 
the  method  and  propose  it  to  the  sponsor  for  approval  or  negotiate  some  of 
the  details. 


Implementation  This  appraisal  method  offers  a  wide  variety  of  choices  that  allow  the 

Guidance  appraisal  team  leader  and  sponsor  to  select  appraisal  features  that  best 

address  appraisal  and  business  objectives. 

Method  tailoring  is  directly  related  to  the  organizational  scope  and 
appraisal  reference  model  scope  decisions.  Most  of  the  allowable  tailoring 
options  flow  logically  from  these  decisions  when  taken  in  context  of  the 
appraisal  objectives  and  constraints.  Tailoring  decisions  typically  affect  the 
appraisal  risk.  Typical  tailoring  choices  that  significantly  impact  appraisal 
planning  include 

•  assigning  mini-teams  by  project  or  by  PA  grouping 

•  data  collection  approaches  to  be  utilized  including  supporting  work 
aids  and  tools  (e.g.,  use  of  video  conference,  teleconference,  or  other 
similar  technology  to  conduct  interviews,  conducting  parallel 
interview  sessions  with  a  minimum  of  two  team  members,  and  use  of 
a  PII  database) 

•  verification  approaches  to  be  utilized,  including  supporting  work  aids 
and  tools  (e.g.,  mini  team  verification  of  practices  at  the  instantiation 
level) 

•  validation  approaches  to  be  utilized  including  supporting  work  aids 
and  tools  (e.g.,  use  of  instrument  or  targeted  focus  group  for 
validation  of  preliminary  findings) 

•  selection  of  optional  SCAMPI  outputs  (e.g.,  preliminary  findings 
focused  on  projects,  division,  or  disciplines,  or  maturity  and/or 
capability  level  ratings) 

•  documenting  strengths  and  non-model  findings 

•  optional  activities  (e.g.,  conduct  executive  session,  plan  for  next  steps, 
or  collect  lessons  learned) 

Experienced  appraisal  team  leaders  will  provide  a  well-defined  approach  to 
ensure  that  the  appraisal  objectives  are  achieved  in  an  efficient  and 
effective  manner.  Experienced  sponsors  will  require  a  well-defined 
approach  to  ensure  an  acceptable  level  of  risk  in  meeting  objectives  within 
the  constraints.  The  appraisal  plan  documents  the  method-tailoring 
decisions  and  their  rationale,  and  the  associated  method  variations  and 
techniques  that  will  be  employed. 
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1.2.2  Identify  Needed  Resources 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


This  activity  is  concerned  with  the  identification  and  estimation  of 
resources  needed  to  carry  out  the  appraisal.  Resources  include  personnel, 
facilities,  tools,  and  access  to  information. 


The  appraisal  team  leader  or  designee  shall 

•  identify  appraisal  team  members 

•  identify  appraisal  participants 

•  identify  equipment  and  facilities 

•  identify  other  appraisal  resources  needed 

•  document  resource  decisions  in  the  appraisal  plan 


The  level  of  detail  in  the  identification  of  needed  resources  must  be 
sufficient  to  support  the  creation  of  the  appraisal  plan.  At  a  minimum,  the 
appraisal  team  leader  must  identify 

•  the  names  of  people  who  are  candidates  for  interviews  or  appraisal 
team  membership 

•  the  organizational  or  project  affiliation  of  these  people 

•  the  location,  seating  capacity,  and  configuration  of  rooms  to  be  used 
by  the  team 

•  specific  equipment  needed  (e.g.,  overhead  projector,  laptop  projector, 
or  video-conferencing) 


Several  months  before  the  appraisal,  tour  the  facility  where  the  appraisal 
will  be  held. 

Assign  an  individual  from  the  appraised  organization  to  carry  out  the 
duties  of  the  organizational  unit  coordinator  (administrative  and  logistical 
support;  see  activity  1.3.2,  Select  Team  Members). 


Con  tin  ued  on  next  page 
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1.2.2  Identify  Needed  Resources  (continued) 


Implementation 

Guidance 


Appraisal  resources  are  typically  defined  early  in  the  appraisal-planning 
process.  Identifying  resources  goes  hand  in  hand  with  estimating  appraisal 
cost  and  schedule  (see  activity  1.2.3,  Determine  Cost  and  Schedule),  and 
these  resources  may  be  iteratively  refined.  Tradeoffs  are  routinely  made  in 
light  of  the  appraisal  objectives  and  constraints. 

The  appraisal  sponsor  or  senior  site  manager  may  identify  candidate 
appraisal  team  members  and  appraisal  participants.  Review  of  the 
organizational  unit  structure  or  other  site-specific  information  can  also  be 
useful  for  this  identification.  Initially,  participants  can  be  specified  in  terms 
of  roles  or  responsibilities,  with  specific  names  to  be  determined  later. 
Process  1.3,  Select  and  Prepare  Team,  contains  additional  guidance  on 
selecting  appraisal  team  members. 

Equipment  and  facilities  are  often  negotiated  with  the  organizational  unit 
where  the  appraisal  activities  will  be  performed,  but  sometimes  these 
equipment  and  facilities  must  be  acquired.  A  room  for  dedicated  use  by  the 
appraisal  team  is  usually  necessary  for  private  discussions  and  to  protect 
the  confidentiality  of  appraisal  data.  Ideally,  this  room  is  separate  from  the 
other  rooms  where  interview  sessions  are  held. 

The  availability  of  computing  resources,  such  as  computers,  printers,  and 
networks,  is  a  key  consideration  that  should  be  planned  and  understood. 
Access  to  special  tools  or  applications  may  also  be  needed. 
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1.2.3  Determine  Cost  and  Schedule 


Activity 

Description 

Required 

Practices 


Parameters 
and  Limits 

Optional 

Practices 


A  top-level  cost  breakdown  and  schedule  are  developed  and  included  in 
the  plan. 


The  appraisal  team  leader  or  designee  shall 

•  estimate  the  duration  of  key  events  as  a  basis  for  deriving  a 
comprehensive  schedule 

•  estimate  the  effort  required  for  the  people  participating  in  the 
appraisal 

•  estimate  the  costs  associated  with  using  facilities  and  equipment  as 
appropriate 

•  estimate  the  costs  for  incidentals  (e.g.,  travel,  lodging,  and  meals)  as 
appropriate 

•  document  a  detailed  schedule  in  the  appraisal  plan 

•  document  detailed  cost  estimates  in  the  appraisal  plan 


Scheduling  for  each  day  of  the  appraisal  is  required. 


None 


Con  tin  ued  on  next  page 
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1.2.3  Determine  Cost  and  Schedule  (continued) 


Implementation 

Guidance 
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Cost  and  schedule  may  be  developed  top  down  based  on  sponsor 
objectives  and  constraints,  bottom  up  based  on  results  of  other  planning 
and  preparation  processes  and  activities,  or  more  generally  using  a 
combination  of  the  two  approaches.  Scheduling  the  events  and  activities  of 
the  appraisal  is  an  ongoing  logistical  task  that  requires  the  coordination  of 
many  different  groups  of  individuals.  Determining  and  communicating  a 
schedule  for  the  appraisal,  and  maintaining  ongoing  visibility  as  the  details 
take  form,  is  the  primary  responsibility  of  the  appraisal  team  leader.  The 
organizational  unit  coordinator  is  expected  to  provide  support  in  this  task, 
and  the  appraisal  team  leader  typically  selects  the  person  who  plays  that 
role  with  this  duty  in  mind. 

The  needs  of  the  sponsor  for  appraisal  outputs  of  a  specified  quality 
fulfilling  a  specified  purpose,  balanced  against  the  resources  available  to 
conduct  the  appraisal,  will  determine  the  schedule  constraints.  Schedule 
and  cost  must  be  considered  for  the  entire  span  of  the  appraisal  activities. 

Effort  estimates  should  be  developed  not  only  for  the  appraisal  team,  but 
also  for  the  expected  participants  within  the  organizational  unit  (e.g., 
interviewees,  respondents  to  instruments  administered,  attendees  at 
briefings,  and  support  staff). 

Organizational  costs  for  preparing  and  supporting  appraisals  can  be 
reduced  by  gathering  and  maintaining  objective  evidence  for  each 
instantiation.  In  addition  to  providing  an  effective  mechanism  for 
monitoring  the  process  implementation  and  improvement  progress  of  the 
organizational  unit,  this  approach  enables  the  ready  availability  and  reuse 
of  objective  evidence  for  subsequent  appraisals. 

While  the  schedule  for  the  appraisal  is  shared  with  a  fairly  wide  audience, 
the  cost  of  the  appraisal  (or  elements  within  the  appraisal)  is  often  kept 
from  wide  view,  due  to  the  potentially  sensitive  nature  of  this  information. 
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1.2.4  Plan  and  Manage  Logistics 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 

Optional 

Practices 


The  logistical  details  of  the  appraisal  are  negotiated  and  documented.  The 
appraisal  team  leader,  supported  by  the  organizational  unit  coordinator, 
manages  planning  tasks  that  document  and  communicate  logistical 
arrangements.  Checklists  and  action  item  tracking  mechanisms  are 
important  structures  used  to  manage  these  tasks. 


The  appraisal  team  leader  or  designee  shall 

•  document  logistical  schedules  and  dependencies 

•  maintain  communication  channels  for  providing  status 

•  assign  responsibilities  for  tracking  logistical  issues 


None 


None 
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1.2.4  Plan  and  Manage  Logistics  (continued) 


Implementation 

Guidance 


Effective  planning  depends  on  anticipating  a  variety  of  logistical  issues 
that  may  occur  during  the  appraisal.  The  time-critical  nature  of  appraisal 
activities  makes  it  difficult  to  manage  last-minute  changes  in  important 
details  such  as  the  following: 

•  identifying  hotels  for  people  traveling  to  the  appraisal 

•  providing  transportation  and/or  lodging  for  team  members  or  the 
remote  members  of  the  organizational  unit 

•  providing  workstation  support 

•  ordering  meals 

•  interacting  with  facilities  staff  on  site 

•  meeting  security/classification  requirements 

•  providing  badges  or  arranging  for  escorts  in  limited-access  facilities 

•  providing  access  to  rooms,  equipment,  and  supplies  needed  for 
administrative  tasks 

•  providing  use  of  virtual  methods  (e.g.,  video  conferences, 
teleconferences,  and  other  similar  technology)  to  conduct  appraisal 
activities 

•  providing  communication  channels  and  back-up  staff  to  support  the 
team  on  site 


If  virtual  methods  such  as  video  conferences,  teleconferences,  and  other 
similar  technology  are  to  be  used  to  perform  appraisal  activities,  these 
methods  should  be  clearly  defined  in  the  Appraisal  Plan.  Furthermore,  it  is 
the  responsibility  of  the  appraisal  team  leader  to  ensure  that  the  use  of 
virtual  methods  in  no  way  compromises  the  integrity  or  accuracy  of  the 
appraisal  activities  or  the  appraisal  results.  Virtual  methods  should  allow 
for  adequate  interaction  between  the  appraisal  team  members  and  the 
appraisal  participants  and  should  provide  mechanisms  for  the  appraisal 
team  to  control  the  interactions. 


♦ 


Page  11-28 


CMU/SEI-2006-HB-002 


1.2.5  Document  and  Manage  Risks 

Activity  As  with  any  project  containing  dependencies  among  events,  people,  and 

Description  other  resources,  risk  management  is  an  important  ingredient  to  success. 

The  appraisal  team  leader  is  responsible  for  documenting  and 
communicating  risks  and  associated  mitigation  plans  to  the  sponsor  and 
appraisal  team  members. 

Required  The  appraisal  team  leader  or  designee  shall 

Practices  •  identify  appraisal  risks 

•  develop  mitigation  plans  for  key  appraisal  risks  and  implement  these 
plans  as  necessary 

•  keep  the  appraisal  sponsor  and  other  stakeholders  informed  of  the 
appraisal  risk  status 

Parameters  None 

and  Limits 

Optional  None 

Practices 

Implementation  The  risks  and  mitigation  plans  identified  through  conducting  this  activity 
Guidance  are  required  elements  of  the  appraisal  plan  (see  Parameters  and  Limits  for 

activity  1.2.6,  Obtain  Commitment  to  Appraisal  Plan).  Most  appraisal  team 
leaders  include  a  section  titled  “Risk  Management”  in  the  appraisal  plan. 
The  level  of  effort  devoted  to  risk-management  activities  is  something  the 
appraisal  team  leader  must  adjust  to  fit  the  situation  at  hand. 

The  appraisal  plan  is  used  to  document  and  track  risks  to  the  successful 
conduct  of  the  appraisal.  As  with  the  requirement  to  address  logistical 
issues  during  planning,  there  are  no  minimum  guidelines  to  be  met  other 
than  the  requirement  that  the  plan  include  identified  risks  and  planned 
mitigation  strategies. 

The  appraisal  team  leader  is  responsible  for  keeping  the  appraisal  sponsor 
informed  of  risk  management  activities  so  that,  if  needed,  timely  sponsor 
intervention  is  possible  to  ensure  the  achievement  of  appraisal  objectives. 
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1.2.6  Obtain  Commitment  to  Appraisal  Plan 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Formal  sponsor  commitment  to  the  appraisal  plan  is  obtained.  The 
appraisal  plan  constitutes  a  contract  between  the  appraisal  sponsor  and  the 
appraisal  team  leader,  so  it  is  vital  that  this  agreement  be  formal. 


The  appraisal  team  leader  or  designee  shall 

•  document  the  appraisal  plan 

•  review  the  appraisal  plan  with  the  sponsor  and  secure  the  sponsor’s 
approval 

•  provide  the  appraisal  plan  to  relevant  stakeholders  for  review 


Required  contents  of  the  appraisal  plan  include  the  following,  at  a 
minimum: 

•  the  appraisal  input  (see  activity  1.1.5,  Obtain  Commitment  to 
Appraisal  Input) 

•  the  activities  to  be  performed  in  conducting  the  appraisal 

•  resources  needed  for  conducting  the  appraisal  (see  activity  1.2.2, 
Identify  Needed  Resources) 

•  cost  and  schedule  estimates  for  performing  the  appraisal  (see  activity 
1.2.3,  Determine  Cost  and  Schedule) 

•  appraisal  logistics  (see  activity  1.2.4,  Plan  and  Manage  Logistics) 

•  risks  and  mitigation  plans  associated  with  appraisal  execution  (see 
activity  1.2.5,  Document  and  Manage  Risks) 

There  must  be  a  signature  block  for  the  appraisal  team  leader  and  the 
sponsor  to  indicate  in  writing  their  commitment  to  the  plan.  If  minor 
updates  are  made  to  the  plan,  signatures  do  not  have  to  be  obtained  again 
except  when  one  or  more  elements  of  the  appraisal  input  have  been 
changed. 

At  a  minimum,  the  appraisal  team  members  are  considered  relevant 
stakeholders  and  should  receive  a  copy  of  the  approved  appraisal  plan. 
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1.2.6  Obtain  Commitment  to  Appraisal  Plan  (continued) 


Optional  Use  a  signature  block  for  relevant  stakeholders  to  indicate  in  writing  their 

Practices  commitment  to  the  plan  (i.e.,  each  team  member  signs  the  plan). 


Implementation  While  sponsor  visibility  into  the  appraisal  plan  is  necessary,  revisions  are 
Guidance  typically  low-level  implementation  details  and  do  not  ordinarily  require 

sponsor  re-approval.  This  low  level  of  change  is  in  contrast  to  the  appraisal 
input,  which  contains  strategic,  key  appraisal  requirements,  objectives,  and 
constraints.  Revisions  to  the  appraisal  input  must  be  approved  by  the 
sponsor.  In  practical  use,  the  appraisal  input  is  often  packaged  as  a 
component  of  the  appraisal  plan,  and  a  single  sponsor  signature  can  serve 
as  approval  for  both.  The  separation  of  the  appraisal  input  and  appraisal 
plan  is  intended  to  provide  an  appropriate  level  of  sponsor  visibility  and 
approval,  while  leaving  appraisal  team  leaders  the  flexibility  to  refine  the 
low-level  details  necessary  to  complete  thorough  appraisal  planning. 
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1.3  Select  and  Prepare  Team 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Ensure  that  an  experienced,  trained,  and  appropriately  qualified  team  is 
available  and  prepared  to  execute  the  appraisal  process. 


•  Appraisal  requirements  have  been  documented  (at  least  in  draft  form). 

•  Appraisal  constraints  are  understood  and  documented  (at  least  in  draft 
form). 

•  The  appraisal  plan  is  defined  (at  least  in  draft  form). 


•  appraisal  requirements  and  constraints  (in  draft  or  final  form) 

•  appraisal  plan  (in  draft  form) 

•  team  training  materials 


1.3.1  Identify  Appraisal  Team  Leader 

1.3.2  Select  Team  Members 

1.3.3  Prepare  Team 


•  training  records 

•  appraisal  team  member  assignments  and  qualifications 

•  a  prepared  appraisal  team  that  has  completed 

-  appraisal  method  training 

-  appraisal  reference  model  training 

-  team-building  activities 

-  team  orientation  regarding  appraisal 


The  successful  completion  of  this  process  results  in  an  experienced, 
trained,  and  oriented  team  ready  to  execute  the  appraisal.  The  appraisal 
team  members  have  acquired  the  necessary  knowledge  to  play  their  roles, 
or  their  previous  knowledge  is  confirmed  to  be  satisfactory.  The  appraisal 
team  leader  has  provided  opportunities  to  practice  the  skills  needed  for 
each  person  to  play  his/her  role,  or  has  confirmed  that  these  skills  have 
already  been  demonstrated  in  the  past.  The  team  members  have  been 
introduced  to  one  another,  and  have  begun  to  plan  how  they  will  work 
together. 


•  The  prepared  team  is  committed  to  the  appraisal. 

•  Training  has  been  provided  and  its  results  recorded. 

•  Remediation  of  knowledge/skill  shortfalls  has  been  completed  (if 
needed). 


Con  tin  ued  on  next  page 
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1.3  Select  and  Prepare  Team  (continued) 


Key  Points  Whether  the  appraisal  team  leader  trains  an  intact  team  or  forms  a  team 

from  a  coips  of  experienced  team  members,  the  responsibility  to  ensure 
that  the  team  is  ready  to  succeed  rests  with  the  appraisal  team  leader. 


Tools  and  Training  course  material  is  available  from  the  CMMI  Steward  for  training 

Techniques  teams.  This  training  should  be  tailored  or  supplemented  by  the  appraisal 

team  leader  based  on  the  appraisal  context  or  degree  of  team  member 
experience.  Case  studies  and  exercises  are  recommended  to  reinforce  the 
situations  team  members  are  likely  to  encounter  during  the  appraisal. 

Other  ways  of  accomplishing  this  activity  may  draw  on  one  or  more  of  the 
following: 

•  providing  supplementary  training  to  previously  experienced  team 
members  so  that  the  operational  details  of  the  approach  used  will  be 
familiar 

•  training  a  cadre  of  team  members  and  keeping  their  knowledge  and 
skills  up-to-date  as  part  of  an  overall  program  of  appraisals 


Metrics 


•  summary  of  team  member  qualifications 

•  effort  and  calendar  time  expended  to  accomplish  training 

•  trainee  ratings  of  instructional  materials  and  approach  (  if  applicable) 

•  achievement  of  milestones  for  remedial  activities  (if  applicable) 


Verification  and 
Validation 


•  sponsor  and  appraisal  team  leader  approval  of  team  membership  and 
preparation 

•  results  of  exams  used  to  demonstrate  training  effectiveness  (if  used) 

•  feedback  from  team  members  on  their  readiness  to  perform  their 
role(s) 


Records 


•  team  member  contact  information 

•  training  records  (if  applicable) 

•  feedback  provided  by  trainees  (if  applicable) 

•  team  qualification  summary  (recorded  in  appraisal  plan) 


Tailoring 


•  Case  study  materials  provide  a  variety  of  options  for  expanding  the 
team  training  course  to  add  emphasis  where  more  is  desired. 

•  Experienced  appraisal  team  leaders  have  had  success  conducting  role- 
plays  and  simulated  appraisal  activities  without  case  studies  as  well. 

•  When  assembling  a  team  of  already-trained  members,  it  is  important 
to  conduct  team-building  activities  to  ensure  team  cohesion.  Many 
team  building  exercises  are  available  for  this  purpose 

•  Team  size,  skills,  and  composition  are  tailoring  options  in  the  method. 


Continued  on  next  page 
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1.3  Select  and  Prepare  Team  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


This  process  includes  selecting  and  preparing  the  appraisal  team.  It  may 
occur  after  obtaining  sponsor  commitment  to  the  appraisal  input.  The 
appraisal  plan  should  be  available,  at  least  in  draft  form,  as  a  necessary 
input  (see  activity  1.2.6,  Obtain  Commitment  to  Appraisal  Plan  for 
contents).  Selected  appraisal  team  members  may  provide  input  into  further 
definition  of  the  appraisal  planning.  Appraisal  team  training  may  provide 
an  initial  means  to  obtain  a  preliminary  understanding  of  the  appraised 
organization’s  operations  and  processes.  If  available,  the  organizational 
unit’s  PII  database  is  a  useful  resource  for  orienting  the  appraisal  team  on 
organizational  characteristics,  such  as  the  application  domain,  the 
organizational  structure,  the  process  improvement  structure,  and 
approaches  for  appraisal  reference  model  implementation. 


The  appraisal  team  is  a  cohesive  unit  of  trained  and  capable  professionals, 
each  of  whom  must  meet  stringent  qualifications.  An  appraisal  team  leader 
is  selected  to  plan  and  manage  the  performance  of  the  appraisal,  delegate 
appraisal  tasks  to  team  members,  and  ensure  adherence  to  SCAMPI  A 
requirements.  Appraisal  team  members  are  selected  based  on  defined 
criteria  for  experience,  knowledge,  and  skills  to  ensure  an  efficient  team 
capable  of  satisfying  the  appraisal  objectives.  Training  is  provided  to 
ensure  proficiency  in  the  appraisal  reference  model  and  appraisal  method. 
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1.3.1  Identify  Appraisal  Team  Leader 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


The  appraisal  sponsor  is  responsible  for  selecting  an  appraisal  team  leader 
who  has  the  appropriate  experience,  knowledge,  and  skills  to  take 
responsibility  for  and  lead  the  appraisal.  By  definition,  an  appraisal  team 
leader  is  a  SCAMPI  Lead  Appraiser,  authorized  by  the  SEI  Appraisal 
Program,  and  is  a  member  of  that  program  in  good  standing.  The  SEI 
Appraisal  Program  is  described  on  the  SEI  Web  site  at 
http://www.sei.cmu.edu/appraisal-program/.  The  appraisal  team  leader  is 
responsible  for  ensuring  that  the  appraisal  is  conducted  in  accordance  with 
SCAMPI  A  requirements,  with  tailoring  to  meet  appraisal  objectives  and 
constraints  within  allowable  bounds  defined  by  the  method. 


The  sponsor  or  designee  shall 

•  select  an  authorized  SCAMPI  Lead  Appraiser  to  serve  as  the  appraisal 
team  leader  (or  a  candidate  SCAMPI  Lead  Appraiser  being  observed 
by  a  Qualified  Observing  Lead  Appraiser) 

•  verify  the  qualifications  of  the  appraisal  team  leader  (experience, 
knowledge,  and  skills) 


The  appraisal  team  leader  must  be  an  SEI-authorized  SCAMPI  Lead 
Appraiser  in  good  standing.  This  authorization  can  be  verified  on  the  Web 
or  by  contacting  the  CMMI  Steward  directly. 

There  can  be  only  one  official  appraisal  team  leader  on  any  given 
appraisal.  The  appraisal  team  leader  has  sole  discretion  to  delegate 
important  tasks  to  appraisal  team  members,  but  cannot  delegate  leadership 
responsibility  or  ultimate  responsibility  for  the  successful  completion  of 
the  appraisal.  The  inclusion  of  multiple  SCAMPI  Lead  Appraisers  on  a 
team  for  a  given  appraisal  can  be  a  strong  asset  for  the  leader  of  that  team. 
However,  the  single  designated  appraisal  team  leader  must  perform  the 
leadership  role  and  manage  the  appraisal  process. 


In  some  uses  of  SCAMPI  A,  representatives  of  the  appraisal  sponsor  may 
perform  a  substantial  part  of  the  appraisal  team  leader’s  tasks  in  advance  of 
the  initial  identification  of  an  appraisal  team  leader.  Infrastructures 
established  to  manage  supplier  selection,  for  example,  may  employ 
standard  acquisition  processes  that  have  well-understood  interfaces  with 
the  appraisal  process. 


Con  tin  ued  on  next  page 
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1.3.1  Identify  Appraisal  Team  Leader  (continued) 


Implementation 

Guidance 
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SCAMPI  Lead  Appraisers,  by  definition,  will  have  participated  on  a 
minimum  of  three  appraisals  (two  as  an  appraisal  team  member  and  one  as 
an  appraisal  team  leader).  These  requirements  are  outlined  in  the  SE1  Lead 
Appraiser  program.  An  additional  consideration  impacting  team 
experience  requirements,  however,  is  the  appraisal  usage  mode  for 
SCAMPI  A.  Additional  experience  may  be  necessary  for  the  appraisal 
team  leader  and/or  appraisal  team  members  if  the  appraisal  is  for  supplier 
selection  and/or  process  monitoring  or  if  it  will  focus  heavily  on  one  of  the 
other  available  disciplines  or  environments,  such  as  acquisition  or 
integrated  product  and  process  development  (IPPD).  Similarly,  if  the 
appraisal  will  be  used  in  a  high  maturity  organization  (maturity  levels  4-5 
or  capability  levels  4-5).  Special  experience,  training,  and/or  expertise 
(e.g.,  statistical  process  control)  may  be  necessary  for  that  type  of 
appraisal. 

Appraisal  team  leader  responsibilities  are  defined  and  described 
throughout  the  SCAMPI  A  MDD,  but  a  summary  overview  of  these 
responsibilities  includes  the  following: 

•  Confirm  the  sponsor’s  commitment  to  proceed  with  the  appraisal. 

•  Ensure  that  appraisal  participants  are  briefed  on  the  purpose,  scope, 
and  approach  of  the  appraisal. 

•  Ensure  that  all  appraisal  team  members  have  the  appropriate 
experience,  knowledge,  and  skills  in  the  appraisal  reference  model 
and  in  SCAMPI  A. 

•  Ensure  that  the  appraisal  is  conducted  in  accordance  with  the 
documented  SCAMPI  A  method. 

•  Verify  and  document  that  the  appraisal  method  requirements  have 
been  met. 

The  appraisal  team  leader  may  be  selected  at  any  time  in  the  appraisal 
planning  phase;  preferably,  the  appraisal  team  leader  is  selected  upon 
initiation  of  appraisal  activities  so  that  he  or  she  may  participate  in 
analyzing  the  requirements  with  the  appraisal  sponsor.  In  any  event,  the 
appraisal  team  leader  should  be  identified  in  time  to  (a)  review  and 
approve  the  appraisal  plan  with  the  appraisal  sponsor  prior  to  beginning  the 
Conduct  Appraisal  phase  of  the  appraisal,  and  (b)  ensure  adequate 
planning  and  the  preparation  of  appraisal  team  members. 
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1.3.2  Select  Team  Members 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


This  activity  involves  identifying  available  personnel,  assessing  their 
qualifications,  and  selecting  them  to  become  appraisal  team  members.  It 
may  occur  after  obtaining  the  sponsor’s  commitment  to  conduct  the 
appraisal  and  may  provide  input  to  appraisal  planning. 


The  appraisal  team  leader  shall 

•  select  individual  team  members  that  meet  the  minimum  criteria  for 
individual  team  members 

•  select  individual  team  members  that  collectively  meet  the  minimum 
criteria  for  the  team  as  a  whole 

•  document  the  qualifications  and  responsibilities  of  team  members  in 
the  appraisal  input 


The  minimum  acceptable  team  size  for  a  SCAMPI  A  appraisal  is  four 
people  (including  the  appraisal  team  leader). 

All  team  members  must  have  previously  completed  the  SEI-licensed 
Introduction  to  CMMI  course. 

With  regard  to  engineering  field  experience,  the  team  (as  a  group)  must 
have  an  average  of  at  least  6  years  of  experience,  and  the  team  total  must 
be  at  least  25  years  of  experience  in  each  of  the  disciplines  to  be  covered  in 
the  appraisal. 

With  regard  to  management  experience,  the  team  (as  a  group)  must  have  a 
total  of  at  least  10  years  of  experience,  and  at  least  one  team  member  must 
have  at  least  6  years  of  experience  as  a  manager. 

The  team  must,  in  aggregate,  have  representative  experience  in  the 
lifecycles  being  appraised. 


Although  not  required  in  the  Parameters  and  Limits  section  above,  the 
following  are  considered  recommended  best  practices  and  should  be 
employed  whenever  feasible: 

•  Each  member  should  have  good  written  and  oral  communication 
skills,  the  ability  to  facilitate  the  free  flow  of  communication,  and  the 
ability  to  perform  as  team  players  and  negotiate  consensus. 

•  At  least  half  of  the  team  members  should  have  participated  in  a 
previous  process  appraisal. 

•  Team  members  should  be  perceived  by  the  appraisal  sponsor  as 
credible. 

Additional  appraisal  team  member  selection  considerations  include 

•  Consider  the  personal  characteristics  of  individual  team  members 
(e.g.,  communication  preferences  and  personality  types)  and  how 
these  characteristics  may  affect  the  dynamics  of  the  team. 

•  Use  one  or  more  authorized  SCAMPI  Lead  Appraisers  as  team 
members. 


Con  tin  ued  on  next  page 


CMU/SEI-2006-HB-002 


Page  11-37 


1.3.2  Select  Team  Members  (continued) 


Implementation 

Guidance 


The  maximum  recommended  team  size  is  nine,  but  a  balance  between  the 
scope  of  the  appraisal  and  the  size  of  the  team  should  be  considered.  Team 
member  training  in  the  appraisal  method  is  discussed  in  activity  1.3.3, 
Prepare  Team. 

Team  members  should  not  be  managers  of  any  of  the  selected  projects  or 
support  groups  or  be  within  the  direct  supervisory  chain  of  any  of  the 
anticipated  interviewees. 

Appraisal  team  members  are  selected  to  provide  a  diverse  set  of  qualified 
professionals  with  the  appropriate  experience,  knowledge,  and  skills  to 
make  reasoned  judgments  regarding  implementation  of  the  appraisal 
reference  model.  The  accuracy  and  credibility  of  the  appraisal  results 
depends  greatly  on  the  capability,  qualifications,  and  preparation  of  the 
appraisal  team  members.  In  addition  to  the  qualifications  described  above, 
other  factors  that  may  affect  the  performance  of  the  team  or  reliability  of 
appraisal  results  should  be  considered.  Appraisal  constraints,  such  as 
security  classification,  may  be  additional  criteria  for  team  member 
selection. 

The  selected  appraisal  team  members  and  their  organizational  affiliation 
and  qualifications  (individually  and  in  aggregate)  are  documented  in  the 
appraisal  plan.  Appraisal  team  members  are  typically  selected  from  a  pool 
of  qualified  individuals  provided  by  the  appraisal  sponsor  or  his/her 
designee.  The  appraisal  team  leader  is  the  final  authority  on  acceptance  of 
appraisal  team  members  and  is  responsible  for  ensuring  their  qualifications 
and  suitability  for  the  appraisal  purpose. 

Situations  where  a  conflict  of  interest  may  arise  should  be  avoided.  Team 
members  who  manage  people  or  processes  in  the  organization  may 
struggle  with  their  ability  to  be  objective.  Team  members  who  are  directly 
impacted  by  the  appraisal  outcome  may  be  distracted  by  the  potential 
consequences  of  the  decisions  they  contribute  to  on  the  appraisal  team. 
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1.3.3  Prepare  Team 


Activity 

Description 


Required 

Practices 


The  appraisal  team  leader  is  responsible  for  ensuring  that  appraisal  team 
members  are  sufficiently  prepared  for  performing  the  planned  appraisal 
activities.  This  preparation  includes  ensuring  team  members  are  familiar 
with  the  appraisal  reference  model,  the  appraisal  method,  the  appraisal 
plan,  organizational  data  and  characteristics,  and  the  tools  and  techniques 
to  be  used  during  the  appraisal.  Roles  and  responsibilities  are  assigned  for 
appraisal  tasks.  Team  building  exercises  are  used  to  practice  facilitation 
skills  and  reach  unity  in  understanding  the  team  objectives  and  how  they 
will  be  satisfied. 

All  team  members  are  expected  to  observe  strict  rules  for  confidentiality, 
the  protection  of  proprietary  or  sensitive  data,  and  the  non-attribution  of 
information  to  appraisal  participants.  Non-disclosure  statements  are  often 
used  to  formalize  these  understandings. 


The  appraisal  team  leader  shall 

•  ensure  that  appraisal  team  members  have  received  appraisal  reference 
model  training 

•  provide  appraisal  method  training  to  appraisal  team  members  or 
ensure  that  they  have  already  received  it 

•  establish  team  building  and  establishing  team  norms 

•  provide  an  orientation  to  team  members  on  appraisal  objectives, 
plans,  and  their  assigned  roles  and  responsibilities 


Continued  on  next  page 
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1.3.3  Prepare  Team  (continued) 


Parameters 
and  Limits 


Model  training  must  be  provided  using  the  standard  Introduction  to  CMMI 
course,  delivered  by  an  instructor  who  is  authorized  by  the  CMMI 
Steward.  At  a  minimum,  all  team  members  must  be  trained  on  the 
following  topics  using  information  from  the  SCAMPI  A  team  training 
materials  provided  by  the  SEI: 

•  SCAMPI  A  method  overview 

•  appraisal  planning,  including  the  contents  of  the  appraisal  plan 

•  objective  evidence  collection  and  analysis 

•  team  decision  making 

•  appraisal  confidentiality  and  non-attribution 

•  practice  characterization 

•  findings  development,  verification,  and  validation 

•  rating 

•  appraisal  output  requirements 

For  teams  involved  in  U.S.  government  source  selection  or  process 
monitoring  appraisals,  team  members  must  also  be  trained  in 

•  applicable  laws,  regulations,  and  policies  that  affect  the  appraisal  such 
as  Federal  Acquisition  Regulations  and  DoD  service  or  organizational 
regulations  and  policies 

•  role  of  the  appraisal  and  the  appraisal  team  in  source  selection  or 
process  monitoring  processes  and  structures 

•  limitations  on  findings  development,  validation,  and  release 

•  special  domain  and/or  model  requirements  (e.g.  space,  command  and 
control,  information  technology,  supplier  sourcing,  and  statistical 
process  management) 

Method  training  delivered  to  groups  of  potential  future  team  members 
must  cover  the  complete  set  of  tailoring  options  and  allowable  variations 
for  the  method  to  prepare  them  for  a  range  of  situations  they  are  likely  to 
encounter  on  future  appraisals.  The  SEI  Appraisal  Program  specifies 
additional  requirements  about  delivering  training  to  people  who  are  not 
already  members  of  an  appraisal  team. 

Team  members  who  have  previously  received  SCAMPI  A  team  training 
are  not  automatically  qualified  to  participate  on  an  appraisal  without  first 
attending  method  training.  In  such  cases,  the  appraisal  team  leader  is 
required  to  understand  the  nature  of  the  training  delivered  previously  and 
the  adequacy  of  that  training  for  the  appraisal  at  hand. 

There  must  be  at  least  one  event  where  the  team  gathers  as  a  group  for  the 
purpose  of  establishing  team  norms  and  make  operational  decisions  about 
how  the  team  will  work  for  the  appraisal  at  hand. 

Due  to  the  confidentiality  required  during  an  appraisal  and  the 
cohesiveness  needed  to  participate  in  appraisal  activities,  observers  are  not 
permitted  to  participate  in  the  appraisal  processes.  The  only  exception  is  an 
observer  who  is  authorized  by  the  CMMI  Steward. 


Continued  on  next  page 
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1.3.3  Prepare  Team  (continued) 

Optional  Some  organizations  have  established  an  “organic”  capability  to  perform 

Practices  appraisals  with  limited  preparation  effort  through  the  use  of  a  pool  of 

trained  appraisal  team  members.  Drawing  from  an  established  group  of 
experts,  who  are  accustomed  to  working  together,  clearly  provides  a 
savings  over  time  for  organizations  that  conduct  frequent  appraisals. 

Implementation  Method  training  may  be  delivered  in  one  of  two  ways: 

Guidance  1 .  method  training  specific  to  the  appraisal  at  hand 

2.  method  training  delivered  to  a  large  group  of  potential  future  team 
members  who  are  not  currently  engaged  in  an  appraisal 

The  team  training  event  is  a  good  place  to  review  the  appraisal  plan  with 
appraisal  team  members,  having  sent  it  to  them  in  advance  of  their  arrival. 
This  event  provides  the  orientation  for  the  entire  appraisal  that  all  appraisal 
team  members  must  execute  their  roles  appropriately.  This  event  also  is  in 
keeping  with  the  “Provide  appraisal  plan  to  relevant  stakeholders  for 
review”  required  practice  in  activity  1.2.6,  Obtain  Commitment  to 
Appraisal  Plan. 

Additionally,  the  team  training  event  is  a  primary  opportunity  to  conduct 
activity  1.5.1,  Perform  Readiness  Review.  The  assembled,  trained 
appraisal  team  can  then  appropriately  assess  the  organization’s  readiness 
for  the  appraisal  and  validate  the  reasonableness  of  appraisal  estimates. 

Implementation  A  typical  model  training  course  is  delivered  in  three  days.  The  successful 
Guidance  completion  of  appraisal  reference  model  training  should  precede  training  in 

the  appraisal  method.  There  is  no  “aging”  requirement  for  when  this  model 
Training  in  training  was  received,  but  the  appraisal  team  leader  is  responsible  for 

the  Reference  ensuring  that  each  team  member  has  adequate  reference  model 

Model  understanding,  and  for  taking  remedial  action  if  necessary.  Attendance  at 

model  training  must  be  recorded  by  the  training  instructor  and  provided  to 
the  CMMI  Steward,  in  accordance  with  the  terms  of  instructor 
authorization. 

For  appraisals  that  include  higher  levels  (i.e.,  maturity/capability  levels  4 
and  5),  team  members  may  benefit  from  receiving  additional  training  on 
this  subject  matter.  The  Intermediate  Concepts  of  CMMI  course,  a  course 
on  Statistical  Process  Control,  and/or  other  advance  topics  may  be  of  use 
for  this  added  level  of  preparation. 

Con  tinued  on  next  page 
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1.3.3  Prepare  Team  (continued) 


Implementation 

Guidance 

Training  in 
the  Appraisal 
Method 


A  typical  delivery  of  appraisal  team  training  might  take  two-and-a-half  to 
three  days.  More  or  less  time  may  be  necessary,  depending  on  the  relative 
experience  of  the  appraisal  team  members. 

Exercises  in  appraisal  techniques  and  team  development  are  used  to 
reinforce  the  skills  that  will  be  important  during  conduct  of  the  appraisal.  It 
is  recommended  that  exercises  be  used  that  are  appropriate  for  the 
organizational  unit  being  appraised.  Where  sufficient  organizational 
artifacts  exist,  “live”  data  can  be  collected  and  used  in  training  exercises 
where  appropriate.  Just-in-time  training  can  also  be  used  to  re-emphasize 
method  concepts  at  appropriate  points  in  the  appraisal  process  during 
which  the  skills  will  be  utilized. 


Appraisal  team  training  materials  should  be  tailored  to  fit  team  needs  and 
objectives  of  the  specific  appraisal.  Tailoring  provides  opportunities  to 

•  provide  insight  into  the  context,  objectives,  and  plans  of  the  particular 
appraisal 

•  communicate  team  members’  assigned  roles  and  responsibilities 

•  identify  tailoring  of  SCAMPI  A  for  the  upcoming  appraisal 

•  acquaint  the  team  with  the  organizational  unit’s  characteristics  and 
documentation 

•  focus  on  skills  that  may  be  more  critical  to  the  upcoming  appraisal, 
such  as  the  ability  to  facilitate  interviews  or  the  ability  to  identify 
alternative  practices 

It  is  recommended  that  this  training  be  provided  within  60  days  of  the 
appraisal.  The  appraisal  team  leader  typically  provides  method  training, 
but  other  delivery  options  are  also  acceptable  (as  described  above). 
Although  alternative  training  options  can  provide  some  advantages  and 
efficiencies  for  method  training,  there  are  also  potential  consequences  that 
might  be  felt  by  the  appraisal  team  leader  on  a  given  appraisal,  such  as 
poor  training  quality  or  inadequate  readiness  of  team  members.  Regardless 
of  how  method  training  is  delivered  to  the  team  members,  opportunities  for 
team  building  should  be  provided  to  coalesce  the  team  and  bring  the  team 
up  to  speed  on  the  specifics  of  the  appraisal  being  planned. 


Implementation 

Guidance 

Familiarization 
with  the 
Appraisal  Plan 


Method  training  and  team  building  provide  good  opportunities  to  establish 
team  familiarity  with  the  appraisal  plan.  This  familiarity  includes  such 
items  as  appraisal  objectives,  organizational  scope,  appraisal  reference 
model  scope,  and  the  schedule,  resources,  and  constraints  for  conducting 
the  appraisal.  Team  member  input  can  be  obtained  to  refine  or  complete 
the  contents  of  the  appraisal  plan. 
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1.3.3  Prepare  Team  (continued) 


Implementation 

Guidance 

Analysis  of 

Objective 

Evidence 


Analysis  of  the  objective  evidence  provided  by  the  appraised  organization, 
such  as  questionnaire  responses  or  worksheets  summarizing  objective 
evidence,  can  be  accomplished  following  or  as  an  integrated  part  of 
appraisal  team  preparation  and  training. 

Demonstrations  or  exercises  using  the  data  collection  tools  and  methods 
planned  for  the  appraisal  should  be  used  to  provide  appraisal  team 
members  with  an  opportunity  to  practice  techniques  for  data  recording, 
verification,  and  analysis.  These  tools  and  methods  may  include 
mechanisms  such  as  wall  charts,  spreadsheets,  or  data  reduction  tools.  The 
more  familiarity  and  comfort  that  can  be  obtained  with  these  tools  in 
advance,  the  greater  the  savings  in  team  efficiency  during  the  Conduct 
Appraisal  phase. 


Con  tin  ued  on  next  page 
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1.3.3  Prepare  Team  (continued) 


Implementation 

Guidance 

Roles  and 
Responsibilities 


The  appraisal  team  leader  should  assign  and  explain  team  member  roles 
and  responsibilities  to  be  performed  during  the  appraisal.  Typical  roles  to 
be  assigned  include 

Organizational  Unit  Coordinator.  The  organizational  unit  coordinator 
handles  logistics  and  provides  technical,  administrative,  and  logistical 
support  to  the  appraisal  team  leader.  This  support  usually  includes 
activities  such  as  coordinating  schedules,  notifying  participants,  arranging 
adequate  facilities  and  resources,  obtaining  requested  documentation,  and 
arranging  catering.  He  or  she  may  also  coordinate  or  provide  clerical 
support  to  the  team.  This  role  is  often  assigned  to  one  or  more  members  of 
the  organizational  unit.  The  organizational  unit  coordinator  may  be  one  of 
the  appraisal  team  members,  or  this  role  may  be  assigned  to  other  site 
personnel. 

Librarian :  The  librarian  manages  the  inventory  of  appraisal  documents, 
coordinates  requests  for  additional  documentation  evidence,  and  returns 
documents  at  the  end  of  the  appraisal.  This  role  can  be  filled  by  an 
appraisal  team  member  or  by  a  member  of  the  support  staff. 

Process  Area  or  Project  Mini-Teams:  Mini-team  members  take  the  lead  for 
data  collection  in  assigned  PAs  or  projects.  They  ensure  that  information 
collected  during  a  data  gathering  session  covers  their  PAs  or  projects, 
request  additional  information  needed  relative  to  their  PAs  or  projects,  and 
record  the  work  performed  by  individual  appraisal  team  members 
pertaining  to  their  PAs  or  projects. 

Mini-teams  typically  consist  of  two  or  three  members.  Mini-team 
assignments  can  be  made  based  on  several  factors,  including 

•  related  PAs  (e.g.,  PA  categories) 

•  composition  mix  of  mini-team  members  (e.g.,  discipline  experience 
and  appraisal  experience) 

Facilitator:  The  facilitator  conducts  interviews. 

Timekeeper:  The  timekeeper  is  responsible  for  tracking  time  and  schedule 
constraints  during  interviews  and  other  activities. 
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1.4  Obtain  and  Inventory  Initial  Objective  Evidence 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Obtain  information  that  facilitates  site-specific  preparation  and  an 
understanding  of  the  implementation  of  model  practices  across  the 
organizational  unit.  Identify  potential  issues,  gaps,  or  risks  to  aid  in 
refining  the  plan.  Strengthen  the  appraisal  team  members’  understanding 
of  the  organization’s  operations  and  processes.  Note:  A  discovery-based 
appraisal  is  a  tailoring  option  of  SCAMPI  A.  If  this  option  is  chosen,  there 
may  be  limited  objective  evidence  to  inventory  at  this  stage  of  the 
appraisal. 


•  appraisal  input  received 

•  sponsor  authorization  to  proceed 

•  availability  of  practice  implementation  data  for  organizational  unit 


•  practice  implementation  data  for  organizational  unit 

•  identified  participants 


1 .4. 1  Obtain  Initial  Objective  Evidence 

1.4.2  Inventory  Objective  Evidence 


•  data  inventory  results  (e.g.,  data  availability  summaries) 

•  identification  of  additional  information  needed 

•  initial  set  of  objective  evidence 


•  Initial  objective  evidence  has  been  collected,  organized,  and  recorded. 

•  Potentially  important  areas  of  needed  information  have  been  noted. 

•  The  team  has  a  deeper  understanding  of  the  organizational  unit’s 
operations  and  processes. 

•  The  team  is  ready  to  make  detailed  plans  for  data  collection. 


•  All  objective  evidence  captured  during  this  activity  has  been  recorded 
for  later  use. 

•  High-priority  areas  for  additional  data  collection  have  been  identified. 

•  The  level  of  sufficiency  of  the  inventory  of  objective  evidence  to 
support  the  appraisal  is  determined. 
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1.4  Obtain  and  Inventory  Initial  Objective  Evidence 
(continued) 


Key  Points  Gather  high-leverage  objective  evidence.  The  amount  of  initial  objective 

evidence  provided  by  the  organization  will  determine  the  proportion  of 
evidence  that  must  be  discovered  (versus  verified)  during  the  appraisal. 
Maximizing  time  spent  in  verification  (versus  discovery)  is  a  key 
performance  objective  for  the  appraisal  process. 


Tools  and 
Techniques 


•  Automated  support,  including  data  reduction  tools,  may  be  available 
to  make  the  data  inventory  activity  more  efficient. 

•  Breaking  into  mini-teams  to  inventory  data  related  to  specific  PAs  is  a 
way  to  help  ensure  completeness  of  the  data. 


Metrics  •  the  number  of  practices  for  which  complete  objective  evidence  is 

available 

•  the  calendar  time  and  effort  expended  for  this  activity  compared  to  the 
planned  values 


Verification  and  •  Where  the  team  includes  members  of  the  appraised  organization, 
Validation  these  members  should  be  used  to  help  understand  the  initial  objective 

evidence  provided  to  prevent  misinterpretation  of  terms  or  special 
conditions. 

•  Inconsistencies  and  contradictions  among  the  items  provided  in  initial 
objective  evidence  should  be  identified  and  recorded  for  resolution. 


Records  •  Lists  of  information  needed  should  be  maintained  and  used  as  input  to 

the  later  data  collection  activities. 

•  Calendar  time  and  effort  expended  in  this  activity  should  be  recorded 
and  compared  to  the  plan.  These  data  will  be  part  of  the  appraisal 
record. 


Tailoring  A  variety  of  methods  can  be  used  to  collect  initial  data,  including 

•  a  site  information  package  prepared  by  representatives  of  the 
organization 

•  a  presentation  on  the  process  improvement  program  and  its 
accomplishments 

•  specialized  or  general  questionnaires  focused  on  practice  implementation 

Note:  In  a  discovery  mode-based  appraisal,  the  above  items  might  not  be 
reviewed  until  after  the  start  of  the  Conduct  Appraisal  phase 

The  use  of  additional  means  of  data  collection  is  dependent  on  the  results 
of  the  inventory  of  available  data  and  the  results  of  process  1.5,  Prepare  for 
Appraisal  Conduct. 


Con  tin  ued  on  next  page 
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1.4  Obtain  and  Inventory  Initial  Objective  Evidence 
(continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


This  process  plays  a  critical  role  in  the  planning  and  preparation  processes. 
The  information  generated  in  this  process  provides  the  most  important 
opportunity  to  reset  expectations  and  plans  with  the  appraisal  sponsor,  if 
initial  assumptions  about  the  availability  of  objective  evidence  turn  out  to 
be  in  error.  It  will  also  provide  the  basis  of  data  collection  planning. 


The  appraisal  team  leader  works  with  representatives  of  the  organization  to 
obtain  an  initial  data  set  that  provides  input  for  an  inventory  of  the 
objective  evidence  pertaining  to  the  implementation  of  each  practice 
among  the  selected  sample  projects  and  support  groups  within  the 
appraisal  scope.  This  initial  data  set  may  be  first  reviewed  by  the  appraisal 
team  leader  for  a  high-level  assessment  of  adequacy  and  completeness. 
The  appraisal  team  leader  or  appraisal  team  then  performs  a  more  detailed 
inventory  to  use  as  input  for  planning  the  data  collection  and  verification 
activities  that  will  occur  when  they  begin  the  Conduct  Appraisal  phase. 
Finally,  a  record  is  created  that  reflects  a  detailed  accounting  of  any 
missing  objective  evidence.  This  record  is  used  as  primary  input  for  the 
generation  of  the  data  collection  plan. 
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1.4.1  Obtain  Initial  Objective  Evidence 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


The  appraisal  team  leader  will  request  that  the  organization  provide 
detailed  data  on  the  implementation  of  practices  in  the  organization.  The 
appraisal  team  leader  is  free  to  specify  the  format  to  be  used  and  the  level 
of  detail  to  be  provided,  knowing  that  anything  that  is  not  provided  in 
advance  must  be  collected  later  in  the  appraisal  process.  There  are  no 
minimum  requirements  set  by  the  method  with  respect  to  completeness  or 
detail  in  this  initial  data  set.  However,  the  effort  required  to  conduct  a 
SCAMPI  A  appraisal  is  a  direct  function  of  the  amount  of  data  available  to 
the  team  at  the  beginning  of  the  process.  Before  the  appraisal  outputs  can 
be  created,  the  team  must  verify  objective  evidence  for  each  instantiation 
of  each  practice  within  the  scope  of  the  appraisal.  For  detailed 
requirements  on  the  sufficiency  of  data,  refer  to  process  2.4,  Verify 
Objective  Evidence. 

The  use  of  a  completely  populated  PII  database  is  desirable  but  not 
essential  at  this  stage  in  the  appraisal  process.  The  appraisal  team  leader 
must  allow  an  opportunity  for  the  organization  to  provide  it,  but  will  not 
require  it  unless  the  sponsor  has  agreed  that  this  appraisal  will  be  a 
verification-based  appraisal  (as  opposed  to  a  discovery-based  appraisal). 


The  appraisal  team  leader  or  designee  shall 
•  obtain  documentation  reflecting  the  implementation  of  model 
practices  among  sample  projects  and  support  groups  within  the 
organizational  unit 


At  a  minimum,  the  organization  must  provide  a  list  of  documents  that  are 
relevant  to  understanding  the  processes  in  use  among  sample  projects  and 
support  groups  within  the  organizational  unit,  unless  a  discovery-based 
appraisal  has  been  planned.  This  list  must  be  mapped  to  the  model 
practices  that  are  in  the  scope  of  the  appraisal. 


A  list  of  terms  and  important  jargon  used  in  the  organizational  unit  may  be 
provided  to  the  team  to  aid  in  communicating  with  the  members  of  the 
organization. 

A  complete  objective  evidence  database,  which  documents  the 
implementation  of  every  model  practice  (within  the  scope  of  the  appraisal) 
among  sample  projects  and  support  groups  within  the  organizational  unit, 
may  be  provided  to  the  team  in  advance. 

The  use  of  database  tools  specifically  built  to  support  a  process  appraisal  is 
highly  recommended. 


Con  tin  ued  on  next  page 
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1.4.1  Obtain  Initial  Objective  Evidence  (continued) 


Implementation 

Guidance 


Whether  collected  using  instruments,  reviewing  documents,  attending 
presentations,  or  conducting  interviews,  the  data  used  for  an  appraisal  is 
related  to  the  practices  of  the  appraisal  reference  model.  For  every  practice 
within  the  reference  model  scope  of  the  appraisal,  and  for  every  instance  of 
each  practice,  objective  evidence  is  used  as  the  basis  for  appraisal  team 
determinations  of  the  extent  to  which  the  practice  is  implemented. 
Indicators  that  substantiate  practice  implementation  include 

•  direct  artifacts,  which  represent  the  primary  tangible  output  of  a 
practice.  These  artifacts  are  typically  listed  in  the  appraisal  reference 
models  as  typical  work  products.  One  or  more  direct  artifacts  are 
necessary  to  verify  the  implementation  of  associated  model  practices. 

•  indirect  artifacts,  which  represent  artifacts  that  are  a  consequence  of 
performing  the  practice,  but  not  necessarily  the  purpose  for  which  it  is 
performed.  These  artifacts  are  typically  things  like  meeting  minutes, 
review  results,  or  written  communications  of  status,  and  may  also  be 
listed  as  typical  work  products. 

•  affirmations,  which  are  oral  or  written  statements  confirming  the 
implementation  of  the  practice.  These  statements  are  typically 
collected  using  interviews,  questionnaires,  or  other  means.  Note: 
Negative  affirmations  confirming  the  lack  of  implementation  of  a 
practice  are  possible. 

Prior  to  the  data  collection  activities  carried  out  by  the  appraisal  team,  an 
initial  data  set  is  usually  created  by  the  appraised  organization.  This  data 
set  contains  descriptions  of  the  objective  evidence  available  for  the  team  to 
examine,  complete  with  references  to  documentation  and  identification  of 
the  personnel  who  can  provide  relevant  affirmations.  The  data  set  provides 
the  baseline  of  objective  evidence  for  the  appraisal.  Most  organizations 
experienced  in  process  improvement  will  already  have  this  type  of  data  on 
hand,  as  they  will  have  used  it  to  track  their  improvement  progress. 

Artifacts  may  be  obtained  as  hard  copies,  soft  copies,  or  hyperlinks  to 
where  these  documents  reside  in  a  Web-based  environment.  If  hyperlinks 
are  used,  the  accessibility  of  artifacts  via  these  links  should  be  verified  in 
the  appraisal  environment.  For  example,  appraisal  team  access  could  be 
inhibited  by  invalid  references  or  firewalls. 

The  initial  data  set  forms  the  basis  for  planning  data  collection  activities, 
including  interviews  and  presentations  on  site.  Any  objective  evidence  that 
is  not  identified  in  advance  of  the  team’s  arrival  must  be  sought  by  the 
team  members  once  they  begin  the  Conduct  Appraisal  phase.  This  process 
of  discovering  whether  and  how  the  organization  has  addressed  a  given 
practice  in  the  model  can  be  quite  time  consuming  and  it  is  often  difficult 
to  predict  how  long  it  will  take. 


♦ 


CMU/SEI-2006-HB-002 


Page  11-49 


1.4.2  Inventory  Objective  Evidence 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


The  inventory  of  the  initial  data  set  provides  critical  new  information  for 
the  overall  planning  of  the  appraisal  and  forms  the  basis  for  the  detailed 
data  collection  plan  that  must  be  developed  before  the  Conduct  Appraisal 
phase.  The  inventory  of  initial  objective  evidence  at  this  stage  is  focused 
primarily  on  the  adequacy  and  completeness  of  information  and  the 
implications  for  future  data  collection.  The  results  of  this  activity  are  the 
primary  basis  for  determining  the  extent  to  which  the  appraisal  will  be  one 
of  verification  or  discovery. 


The  appraisal  team  leader  and/or  designees  shall 

•  examine  the  initial  set  of  objective  evidence  provided  by  the 
organizational  unit,  unless  a  discovery-based  appraisal  has  been 
selected 

•  determine  the  extent  to  which  additional  information  is  needed  for 
adequate  coverage  of  model  practices 


Information  provided  by  the  organizational  unit  must  be  detailed  enough  to 
understand  the  extent  to  which  each  type  of  objective  evidence  (i.e.,  direct 
artifacts,  indirect  artifacts,  and  affirmations)  is  available  for  each  process 
instantiation  for  each  model  practice  within  the  scope  of  the  appraisal.  This 
initial  review  of  objective  evidence  identifies  model  practices  for  which 
the  team  may  decide  it  has 

•  strong  objective  evidence 

•  no  objective  evidence 

•  conflicting  objective  evidence 

•  inconsistent  objective  evidence 

•  insufficient  objective  evidence 

In  the  process  of  inventorying  the  available  objective  evidence,  potential 
alternative  practices  must  be  considered.  Any  objective  evidence  for  such 
practices  must  be  determined  as  early  as  possible.  See  Appendix  C, 
Alternative  Practice  Identification  and  Characterization  Guidance,  for 
information  on  identifying  acceptable  alternative  practices. 

Key  documents  are  identified  that  can  be  used  to  gain  insight  regarding  a 
number  of  model  practices.  These  documents  contain  potential  high- 
leverage  information  and  may  be  good  candidates  for  early  review  by  team 
members. 


Review  the  initial  objective  evidence  with  members  of  an  engineering 
process  group  and/or  project  representatives. 


Con  tin  ued  on  next  page 
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1.4.2  Inventory  Objective  Evidence  (continued) 


Implementation 

Guidance 


Members  of  the  team  may  choose  to  summarize  the  extent  of  available 
information  available  at  the  discretion  of  the  appraisal  team  leader. 
However,  the  objective  of  this  activity  is  to  determine  how  much 
additional  data  team  members  must  gather  to  complete  their  work.  It  is 
recommended  that  the  appraisal  team  leader  establish  an  expectation  with 
the  sponsor  that  the  results  of  this  activity  will  form  the  basis  for  a  revised 
schedule  estimate.  If  the  initial  objective  evidence  is  lacking  in 
completeness  and  detail,  the  team  will  be  forced  to  seek  more  information 
during  the  Conduct  Appraisal  phase,  unless  corrective  actions  are  taken 
before  that  time. 

It  is  important  to  keep  all  stakeholders  focused  on  the  fact  that  SCAMPI  A 
is  intended  as  a  benchmarking  appraisal.  This  method  is  not  well  suited  for 
organizations  that  have  a  limited  understanding  of  CMMI.  Such 
organizations  may  not  yet  have  a  clear  idea  of  how  the  practices  described 
in  the  reference  models  ought  to  be  implemented  to  meet  their  specific 
business  needs.  Deciding  on  a  reasonable  implementation  of  the  practices, 
and  working  to  ensure  that  they  are  enacted  throughout  the  organization, 
are  activities  that  precede  a  benchmarking  appraisal.  A  different  type  of 
appraisal  (Class  B  or  C)  is  probably  more  valuable  if  the  objective  of  the 
sponsor  is  to  begin  the  process  of  understanding  what  CMMI  could  mean 
for  the  organization.  It  is  not  reasonable  to  schedule  a  two-week  appraisal 
and  expect  to  collect  all  of  the  data  required  for  benchmarking  during  the 
Conduct  Appraisal  phase. 

The  appraisal  team  leader  often  reviews  the  initial  data  set  provided  by  the 
organization  prior  to  assembling  the  team  for  its  first  meeting  to  identify 
areas  where  additional  data  will  be  needed  and  to  assess  the  feasibility  of 
the  planned  appraisal  schedule.  This  readiness  review  should  be  conducted 
prior  to  finalizing  the  appraisal  schedule,  and  may  comprise  a  “go/no-go” 
decision  for  the  appraisal  in  some  situations.  The  appraisal  team  may  then 
review  the  initial  objective  evidence  in  more  detail  (typically  toward  the 
end  of  the  team-training  event)  to  begin  formulating  plans  for  how  missing 
evidence  will  be  collected,  and  to  verify  the  entire  data  set.  This 
preliminary  readiness  review  is  the  basis  for  the  data  collection  plan,  which 
is  described  in  the  next  process,  1.5,  Prepare  for  Appraisal  Conduct. 
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1.4.2  Inventory  Objective  Evidence  (continued) 


Implementation 

Guidance 

(continued) 


The  appraisal  team  leader  generates  a  list  of  additional  information  needed. 
The  results  of  the  inventory  of  initial  objective  evidence  are  documented  as 
an  input  to  the  data  collection  plan.  The  use  of  an  integrated  appraisal  tool 
to  annotate  the  set  of  initial  objective  evidence  will  permit  the  automated 
tracking  of  information  needs,  and  will  aid  in  the  compilation  of  a  detailed 
data  collection  plan.  Where  the  completeness  of  initial  objective  evidence 
is  insufficient  to  conduct  the  appraisal  under  the  original  schedule,  the 
results  of  this  activity  form  an  important  basis  for  renegotiating  the 
appraisal  schedule  in  some  cases. 

The  adequacy  of  objective  evidence  relative  to  model  practices  is  typically 
determined  using  a  software  tool  of  some  sort,  either  one  built  for  use  on 
appraisals  or  a  spreadsheet  template.  However,  paper  forms  and  wall  charts 
may  be  used  if  preferred. 
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1 .5  Prepare  for  Appraisal  Conduct 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Ensure  readiness  to  conduct  the  appraisal,  including  confirmation  of  the 
availability  of  objective  evidence,  appraisal  team  commitment,  logistics 
arrangements,  risk  status  and  associated  mitigation  plans.  Plan  and 
document  data  collection  strategies. 


•  Sponsor  commitment  to  proceed  with  the  appraisal  has  been 
documented. 

•  Appraisal  objectives  and  constraints  have  been  documented. 

•  Initial  objective  evidence  has  been  received  and  an  inventory  has  been 
completed. 

•  Appraisal  logistics  and  risks  have  been  documented. 

•  The  appraisal  team  is  trained. 


•  appraisal  plan 

•  PIIs  for  the  organizational  unit 

•  initial  objective  evidence  inventory 

•  data  collection  status 


1.5.1  Perform  Readiness  Review 

1.5.2  Prepare  Data  Collection  Plan 

1.5.3  Replan  Data  Collection 


•  initial  data  collection  plan 

•  updates  to  the  data  collection  plan  as  required 


•  updated  plans 

•  team  member  awareness  of  data  status  and  needs 


•  The  team  is  ready  to  conduct  the  appraisal. 

•  Logistical  arrangements  are  confirmed. 

•  All  preparations  for  data  collection  by  the  team  have  been  made  and 
the  data  collection  plan  has  been  documented. 


Con  tin  ued  on  next  page 
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1 .5  Prepare  for  Appraisal  Conduct  (continued) 


Key  Points  Performance  of  at  least  one  readiness  review  resulting  in  the  appraisal  team 

leader’s  and  sponsor’s  joint  decision  to  continue  the  appraisal  as  planned, 
re-plan  the  appraisal,  or  cancel  the  appraisal  is  paramount  to  the  successful 
conduct  of  the  appraisal.  The  data  collected  are  the  most  important  input 
the  team  receives.  Careful  planning,  disciplined  tracking  against  the  plan, 
and  effective  corrective  actions  are  cornerstones  to  success  in  this  process. 

Tools  and  The  use  of  a  spreadsheet  to  record  and  track  the  data  collection  plan  is  a 

Techniques  common  technique.  A  matrix  showing  the  practices  of  the  model,  or 

questions  to  be  asked,  arrayed  on  the  vertical  axis  and  the  sources  of 
information  arrayed  on  the  horizontal  axis  provides  a  simple  planning  and 
tracking  tool.  A  number  of  vendor-provided  data  management  tools  are 
available  as  well. 

Metrics  •  estimated  and  tracked  calendar  time  and  effort  for  this  activity 

•  planned  and  actual  number  of  data  sources  per  practice 

•  planned  and  tracked  number  of  scripted  questions  used  per  interview 

•  planned  and  tracked  number  of  scripted  questions  used  per  PA 

•  percentage  of  planned  coverage  achieved  per  data  collection  event  or 
PA 

•  number  of  changes  to  the  appraisal  plan  relative  to  the  appraisal  team, 
logistics,  and  risks 

Verification  and  The  data  collection  plan  should  be  summarized  and  reviewed  with  the  team 

Validation  to  ensure  that  appraisal  requirements  will  be  successfully  implemented  if 

the  plan  is  carried  forward.  Experienced  appraisal  team  leaders  will  use 
historical  data  to  assess  the  feasibility  of  (and  risks  associated  with)  the 
data  collection  plan. 

Records  Planned  and  actual  coverage  of  practices  and  PAs  across  the  set  of  data 

collection  activities  should  be  recorded.  These  data  support  future 
estimates  and  corrective  actions  during  data  collection  activities. 

Tailoring  Replanning  is  performed  when  the  status  of  the  appraisal  indicates  the  need 

to  do  so. 

Additional  planning  and  coordination  steps  may  be  necessary  in  usage 
modes  in  which  data  collection  activities  will  occur  at  geographically 
distributed  sites  or  across  organizational  units  from  different  corporations 
(e.g.,  in  a  supplier  selection  usage  mode). 

SCAMPI  A  allows  great  flexibility  in  formulating  strategies  to  accomplish 
the  necessary  data  collection.  The  relative  emphasis  of  different  data 
sources,  as  well  as  data  types,  can  be  tuned  to  support  appraisal  objectives 
relating  to  buy-in  as  well  as  coverage  and  rigor  for  important  areas. 

Continued  on  next  page 
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1 .5  Prepare  for  Appraisal  Conduct  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  data  collection  plan  is  an  essential  element  of  the  appraisal  plan.  The 
activities  described  here  rely  on  the  results  of  an  inventory  of  the  initial 
objective  evidence  to  derive  a  plan  and  set  of  strategies  for  accomplishing 
the  data  collection  needed  to  meet  the  objectives  of  the  appraisal.  The  data 
collection  plan  developed  through  these  activities  is  reviewed  and  revised 
on  a  continual  basis  throughout  the  appraisal.  Dynamically  managing  the 
inventory  of  data  on  hand,  the  list  of  data  needed,  and  the  available  data 
collection  opportunities  are  processes  critical  to  the  success  of  the 
appraisal. 


The  activities  in  this  process  serve  to  (a)  ensure  readiness  to  conduct  the 
appraisal,  (b)  establish  the  initial  planning  baseline  for  the  acquisition  of 
objective  evidence,  and  (c)  update  the  plan  to  account  for  information 
acquired  and  unexpected  developments.  Since  SCAMPI  A  is  a  data- 
intensive  method,  the  conduct  of  these  activities  in  accordance  with  the 
descriptions  provided  is  essential  to  the  successful  use  of  the  appraisal 
method. 
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1.5.1  Perform  Readiness  Review 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


The  purpose  of  the  readiness  review  is  to  determine  whether  or  not  the 
appraisal  team  and  appraised  organization  are  ready  to  conduct  the 
appraisal  as  planned.  The  readiness  review  addresses  several  aspects  of 
readiness  to  conduct  the  appraisal:  data  readiness,  team  readiness,  logistics 
readiness,  and  appraisal  risk  status.  The  readiness  review  will  result  in  a 
decision  to  continue  as  planned,  re-plan  or  reschedule,  or  cancel  the 
appraisal.  The  appraisal  team  leader  and  sponsor  are  responsible  for  the 
decision  and  determining  the  conditions  under  which  to  proceed. 


The  appraisal  team  leader  or  designee  shall 

•  determine  whether  the  objective  evidence  for  each  instance  of  each 
practice  in  the  appraisal  scope  is  adequate  to  proceed  with  the 
appraisal  as  planned  (refer  to  activity  1.4.2,  Inventory  Objective 
Evidence) 

•  determine  whether  the  appraisal  team  is  prepared  to  conduct  the 
appraisal  (refer  to  activity  1.3.3,  Prepare  Team) 

•  ensure  the  appraisal  logistics  (e.g.  facilities,  equipment,  and 
participant  availability)  have  been  arranged  and  confirmed  (refer  to 
activity  1.2.4,  Plan  and  Manage  Logistics) 

•  review  identified  appraisal  risks  to  determine  status  and  impact  to 
conducting  the  appraisal  as  planned  (refer  to  activity  1.2.5,  Document 
and  Manage  Risks) 

•  review  the  feasibility  of  the  appraisal  plan  in  light  of  data  readiness, 
team  readiness,  logistics  readiness,  and  overall  risk 


Explicit  criteria  for  determining  readiness  must  be  established  by  the 
appraisal  team  leader. 

At  least  one  readiness  review  must  be  conducted. 

If  any  practice  characterizations  are  done  before  or  during  the  readiness 
review,  then  the  Conduct  Appraisal  phase  has  begun  and  must  be 
completed  within  the  90-day  constraint. 


Integrating  a  readiness  review  with  the  team  training  event  will  allow  the 
appraisal  team  leader  to  involve  the  team  in  gaining  an  understanding  of 
the  data  available  to  support  the  appraisal. 

The  readiness  review  may  be  conducted  on-site,  by  video-  or  tele¬ 
conference,  or  by  any  combination  of  these. 


Con  tin  ued  on  next  page 
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1.5.1  Perform  Readiness  Review  (continued) 


Implementation 

Guidance 


More  than  one  readiness  review  might  be  needed.  One  should  be 
performed  early  in  the  planning  phase,  long  enough  in  advance  to  give  the 
organization  time  to  collect  any  additional  objective  evidence  and  for  the 
appraisal  team  leader  to  address  any  logistics  problems,  team  issues,  or 
other  critical  appraisal  risks  to  support  a  more  successful  appraisal. 
Another  should  be  performed  once  the  objective  evidence  has  been 
gathered  and  the  appraisal  is  ready  to  start.  This  review  may  be  conducted 
in  conjunction  with  the  team  training  event. 

The  readiness  review  should  be  led  by  the  appraisal  team  leader. 
Recommended  participants  include  at  least  one  representative  from  each 
appraisal  mini-team,  the  organizational  unit  coordinator,  and  any 
additional  organizational  unit  representatives  desired. 

Data  readiness  should  address  what  data  is  available,  what  data  is  still 
needed,  and  how  and  where  additional  data  will  be  obtained. 

Recommended  minimum  criteria  for  data  readiness  include 

•  PII  databases  do  not  have  any  significant  coverage  gaps. 

•  Artifacts  identified  in  the  PII  databases  are  accessible. 

•  The  state  of  readiness  and  completeness  is  consistent  with  the 
duration  of  the  planned  Conduct  Appraisal  phase. 

Thresholds  for  the  sufficiency  of  data  should  be  established  as  targets  to  be 
met  at  the  readiness  review.  For  example,  an  80%  threshold  may  be  used  to 
initiate  replanning  at  the  final  readiness  review.  That  is,  the  appraisal  team 
leader  establishes  an  expectation  with  the  sponsor  that,  if  more  than  20% 
of  the  objective  evidence  is  missing  at  the  time  of  team  training,  the 
appraisal  must  be  replanned.  However,  the  primary  objective  is  reducing 
the  risk  that  there  will  be  insufficient  objective  evidence  to  make  the 
determinations  required  by  the  appraisal  plan  in  the  time  allotted. 

Objective  evidence  for  all  projects  and  support  groups  sampled  to 
represent  the  organizational  unit  should  be  reviewed. 

Objective  evidence  for  alternative  practices  should  be  considered.  See 
Appendix  C,  Alternative  Practice  Identification  and  Characterization 
Guidance,  for  information  on  identifying  acceptable  alternative  practices. 

Team  readiness  should  address  whether  the  appraisal  team  is  prepared  to 
conduct  the  appraisal.  The  appraisal  team  leader  should  determine  whether 
the  team  members  are  adequately  trained  and  the  mini-teams  are  operating 
effectively  and  efficiently.  If  necessary,  the  appraisal  team  leader  may 
change  mini-team  membership,  add  resources,  or  change  assignments  to 
optimize  team  performance. 
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1.5.1  Perform  Readiness  Review  (continued) 

Implementation  Logistics  readiness  should  address  whether  the  necessary  logistics 

Guidance  arrangements  have  been  made  for  the  Conduct  Appraisal  phase.  The 

(continued)  appraisal  team  leader  should  review  the  logistics  arrangements  to 

determine  whether  appropriate  facilities  have  been  reserved,  the  necessary 
equipment  will  be  available,  and  the  appraisal  participants  have  been 
contacted  and  will  be  available  during  the  Conduct  Appraisal  phase. 

A  summary  of  the  inventory  of  objective  evidence  and  readiness  to 
proceed  should  be  reviewed  with  the  sponsor  or  his/her  designee.  If 
insufficient  objective  evidence  is  available  or  if  any  other  aspect  of 

appraisal  readiness  is  not  met,  the  appraisal  team  leader  may  need  to 

initiate  replanning  in  light  of  newly  discovered  constraints  (e.g., 
insufficient  data  to  support  the  appraisal  as  planned).  Refer  to  activity 
1.1.2,  Determine  Appraisal  Constraints.  The  criteria  for  adequacy  will 
depend  on  where  the  readiness  review  occurs  in  the  schedule,  and  the 
degree  of  verification  versus  discovery  that  is  being  sought  for  the  Conduct 
Appraisal  phase  of  the  appraisal. 

The  readiness  review  is  a  key  event  whose  impact  should  not  be 
underestimated.  Failure  to  ensure  that  all  aspects  of  appraisal  readiness 
(i.e.,  data,  team,  logistics,  and  overall  risk)  are  reviewed  to  determine  the 
impact  on  the  appraisal  plan  can  have  grave  consequences  for  the  appraisal 
team  during  the  Conduct  Appraisal  phase.  The  Conduct  Appraisal  phase 
may  include  long  hours,  exhaustion,  extensive  ad  hoc  data  collection  (i.e., 
discovery),  or  the  inability  to  achieve  appraisal  objectives  within  defined 
estimates  and  constraints. 


♦ 
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1.5.2  Prepare  Data  Collection  Plan 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  data  collection  activities  are  tailored  to  meet  the  needs  for  objective 
evidence  so  that  the  extent  of  practice  implementation  can  be  determined. 

For  practices  that  have  objective  evidence,  a  strategy  for  verifying  that 
evidence  will  be  formulated. 

For  practices  that  lack  objective  evidence,  a  strategy  for  discovering  that 
evidence  will  be  formulated. 

The  data  collection  plan  is  typically  embodied  in  a  number  of  different 
artifacts  used  during  the  appraisal  process.  The  appraisal  plan  includes 
information  about  the  site,  sample  projects  and  support  groups,  and 
participants  involved  in  the  appraisal.  This  plan  information  is  high  level 
information  that  helps  document  and  communicate  the  data  collection  plan. 
Detailed  information  on  data  collection  can  be  recorded  in  work  aids  that 
manage  appraisal  data  and  in  the  appraisal  schedule.  A  record  of 
“information  needed”  items  is  the  most  detailed,  while  document  lists, 
interview  schedules,  and  the  assignment  of  PA  mini-teams  help  shape  the 
strategy  for  obtaining  the  needed  data. 


The  appraisal  team  leader  or  designee  shall 

•  determine  participants  for  interviews 

•  determine  artifacts  to  be  reviewed 

•  determine  presentations/demonstrations  to  be  provided 

•  determine  team  roles  and  responsibilities  for  data  collection  activities 

•  document  the  data  collection  plan 


The  data  collection  plan  must  specify  contingencies  to  manage  the  risk  of 
having  insufficient  data. 

For  every  instantiation  of  every  model  practice,  the  data  collection  plan 
must  specify  how,  when,  and  by  whom  the  objective  evidence  will  be 
verified. 

For  instantiations  of  model  practices  that  have  not  been  addressed  in  the 
initial  objective  evidence,  the  data  collection  plan  must  specify  how  the 
team  intends  to  discover  the  presence  or  absence  of  objective  evidence  that 
characterizes  the  extent  of  implementation  for  that  practice. 


Continued  on  next  page 
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1.5.2  Prepare  Data  Collection  Plan  (continued) 


Parameters 
and  Limits 
(continued) 


Optional 

Practices 


Page  11-60 


The  data  collection  plan  (often  documented  in  a  variety  of  artifacts) 
includes 

•  assignment  of  PAs  to  team  members 

•  a  summary  of  initial  objective  evidence  provided  by  the  organization 

•  identification  of  highest  priority  data  needs 

•  initial  allocation  of  data  must  data-gathering  events 

•  identification  of  instruments  to  be  administered,  if  any 

•  identification  of  participants  to  be  interviewed 

•  an  interview  schedule,  revised  to  include  more  detail 

•  identification  of  a  starter  set  of  interview  questions 

•  identification  of  documents  still  needed  (if  any) 

•  risks  associated  with  the  sufficiency  of  the  data  and  the  adequacy  of 
the  schedule 


Review  the  status  of  the  objective  evidence  database  with  members  of  the 
appraised  organization  to  elicit  additional  objective  evidence  or  to  expand 
on  the  evidence  available.  This  review  allows  the  appraisal  team  leader  to 
validate  the  data  collection  plan  to  some  extent. 


Continued  on  next  page 


CMU/SEI-2006-HB-002 


1.5.2  Prepare  Data  Collection  Plan  (continued) 


Implementation 

Guidance 


Types  of  objective  evidence  include  documents  and  interviews  (see 
process  2.2,  Examine  Objective  Evidence).  Objective  evidence  is 
differentiated  in  terms  of  different  types  of  PIIs  (i.e.,  direct  artifacts, 
indirect  artifacts,  and  affirmations),  as  described  in  activity  1.4.1,  Obtain 
Initial  Objective  Evidence.  A  combination  of  these  indicator  types  is 
required  for  corroboration  (see  activity  2.4.1,  Verify  Objective  Evidence). 
The  data  collection  status  is  continually  monitored  during  appraisal 
activities  (see  process  2.3,  Document  Objective  Evidence)  to  ensure  that 
sufficient  data  coverage  is  obtained.  These  key  considerations  should  be 
understood  and  accounted  for  in  the  generation  of  the  data  collection  plan. 

Multiple  types  of  interviews  can  be  used  to  obtain  oral  affirmations  (see 
activity  2.2.2,  Examine  Objective  Evidence  from  Interviews): 

•  standard  structured  interviews  scheduled  in  advance  that  use  scripted 
questions 

•  on-call  interviews,  scheduled  in  advance  for  calendar  purposes,  but 
held  only  if  it  is  determined  they  are  necessary 

•  office  hours  interviews  in  which  interviewees  are  notified  that  they 
may  need  to  be  available  as  a  contingency  during  scheduled  periods 

The  data  collection  plan  should  specify  sources  of  data,  tools,  and 
techniques  to  be  used. 

A  robust  data  collection  plan  will  plan  for  interviews  of  all  three  types. 
Start  with  a  full  set  of  scheduled  interviews  and  gradually  add,  eliminate, 
or  modify  events  as  the  inventory  of  initial  objective  evidence  indicates  the 
need. 

The  data  collection  plan  should  clearly  specify  whether  any  virtual 
methods  (e.g.,  video  conferences,  teleconferences,  and  other  similar 
technology)  will  be  used  and  how  they  will  be  used.  It  is  the  responsibility 
of  the  appraisal  team  leader  to  ensure  that  virtual  methods  do  not 
compromise  the  integrity  or  accuracy  of  appraisal  activities  or  appraisal 
results. 

Planning  for  document  reviews  should  include  organization-level,  project- 
level,  and  implementation-level  artifacts,  as  described  in  activity  2.2.1, 
Examine  Objective  Evidence  from  Documents. 

Ultimately,  the  appraisal  team  must  have  data  on  each  practice  in  the 
reference  model  within  the  appraisal  scope  for  each  organizational  element 
within  the  appraisal  scope.  For  PAs  addressing  practices  implemented  at 
the  project/program  level  (e.g.,  Project  Planning),  this  requirement  means 
that  data  on  each  instantiation  of  the  practice  will  be  collected.  For  PAs 
addressing  practices  implemented  at  the  organization  level  (e.g., 
Organizational  Training),  only  one  instantiation  of  each  practice  may  be 
needed,  depending  on  the  way  the  organization  chooses  to  implement  such 
practices. 
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1.5.2  Prepare  Data  Collection  Plan  (continued) 


Implementation 

Guidance 

(continued) 


Page  11-62 


The  results  of  the  analysis  of  initial  objective  evidence  are  used  to 
determine  which  practices  are  not  already  covered  with  objective  evidence. 
Practices  for  which  no  initial  objective  evidence  has  been  provided  should 
be  identified  as  high-risk  areas  for  the  team  to  address  immediately.  The 
schedule  for  data  collection  may  need  to  change  dramatically  if  the  team  is 
unable  to  find  relevant  data  for  these  areas  in  short  order.  In  the  case  of 
practices  for  which  data  are  available  in  the  initial  objective  evidence,  the 
team  members  assigned  to  the  PAs  plan  the  strategy  for  verifying  the 
implementation  of  each  of  the  practices  through  review  of  the  named 
documents,  interviews  with  the  people  who  fill  the  named  roles,  or  other 
data  collection  events.  Artifacts  used  to  manage  data  collection  events  are 
populated  with  the  current  understanding  of  the  planned  data  collection 
events,  as  follows: 

•  The  schedule  for  interviews  is  finalized,  so  participants  can  be 
informed  of  the  expectations  for  their  participation  as  interviewees. 

•  The  list  of  documents  on  hand  (e.g.,  accessible  electronically)  is 
finalized  so  that  the  team  members  know  what  is  and  is  not  available 
for  document  review. 

•  A  preliminary  allocation  of  practices  to  be  covered  in  each  of  the 
scheduled  interviews  is  documented. 

•  A  list  of  needed  documents  (not  yet  available  to  the  team)  is 
generated,  if  there  are  any  known  needs  for  documents  at  this  point. 
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1.5.3  Replan  Data  Collection 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  data  collection  plan  is  updated  as  required  during  the  conduct  of  the 
readiness  review  or  during  the  appraisal  itself  as  objective  evidence  is 
found,  or  as  new  sources  of  information  are  uncovered.  The  activity 
described  in  this  section  refers  to  a  more  substantial  change  in  the  plan, 
which  is  expected  to  be  a  rare  occurrence  in  practice.  If  during  the  conduct 
of  an  appraisal  the  team  discovers  that  their  assumptions  about  the 
availability  of  objective  evidence  are  substantially  incorrect,  the  appraisal 
team  leader  may  renegotiate  the  appraisal  plan  with  the  sponsor. 


The  appraisal  team  leader  or  designee  shall 

•  review  the  current  inventory  of  objective  evidence  and  determine 
model  practices  for  which  the  objective  evidence  is  inadequate 
relative  to  the  appraisal  plan 

•  revise  the  data  collection  plan  as  necessary  based  on  the  appraisal 
status  and  availability  of  objective  evidence 

•  renegotiate  the  appraisal  plan  with  the  sponsor  if  the  appraisal  cannot 
proceed  as  planned 


None 
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1.5.3  Replan  Data  Collection  (continued) 


Optional  A  risk  analysis  can  be  conducted  during  early  planning  activities  to 

Practices  establish  thresholds  and  limits  for  the  amount  of  missing  objective 

evidence  that  will  trigger  this  activity.  This  analysis  enables  the  appraisal 
team  leader  to  state,  in  advance,  the  conditions  under  which  the  team  and 
the  sponsor  renegotiate  the  appraisal  plan. 

Contingency  planning  done  in  advance  to  identify  ways  of  overcoming 
issues  associated  with  missing  objective  evidence  could  include 

•  an  alternate  (fail-back)  schedule  for  the  appraisal 

•  staffing  to  conduct  a  “crash  data  collection”  activity 

•  reducing  the  scope  of  the  appraisal  (e.g.,  appraising  fewer  PAs  and/or 
limiting  the  extent  of  the  organizational  unit  appraised) 


Implementation  This  activity  is  not  a  substitute  for  tactical  decisions  about  where  and  how 

Guidance  to  find  objective  evidence.  The  intent  of  this  activity  is  to  respond  to  a 

major  gap  between  expected  data  and  actual  data. 

Major  gaps  between  expected  and  actual  data  may  occur,  for  example,  as  a 
result  of  the  following: 

•  inaccurate  assumptions  about  the  availability  of  objective  evidence 

•  content  of  artifacts  or  information  from  interviews  not  providing 
significant  amounts  of  the  information  required  and  other  sources  not 
being  planned 

•  unexpected  absence  of  multiple  key  interviewees 

•  unanticipated  delays  in  the  implementation  of  new  processes 

•  major  customer-driven  emergencies  for  one  or  more  of  the  sampled 
projects  or  support  groups 

This  activity  serves  as  a  “pressure  valve”  of  sorts  for  the  appraisal.  The 
pressure  to  perform  the  appraisal  under  unrealistic  conditions  can  lead  to  a 
severe  degradation  in  the  quality  of  the  appraisal  outputs.  Carefully 
planning  for  contingencies  and  communicating  them  to  the  sponsor  help  to 
protect  the  standards  that  must  be  met  in  the  performance  of  an  appraisal. 
Clearly  documenting  the  data  collection  plan,  and  regularly  monitoring  the 
availability  of  data  compared  to  that  plan,  support  effective  risk  mitigation. 

When  this  activity  must  be  employed  to  recover  from  an  unrealistic 
expectation,  the  documentation  reflecting  the  assumptions  made  during 
planning,  as  well  as  concrete  facts  about  what  is  or  is  not  available,  are 
used  to  renegotiate  with  the  appraisal  sponsor.  This  need  to  renegotiate  is 
one  of  the  reasons  why  a  detailed  appraisal  plan,  with  the  sponsor’s 
signature,  is  a  required  artifact  for  the  conduct  of  a  SCAMPI  A  appraisal. 


♦ 
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2  Conduct  Appraisal 


2.1  Prepare  Participants 


Purpose 

Ensure  that  appraisal  participants  are  appropriately  informed  of  the 
appraisal  process,  purpose,  and  objectives  and  are  available  to  participate 
in  the  appraisal  process. 

Entry  Criteria 

•  Data  collection  has  been  planned. 

•  The  sponsor  has  approved  the  appraisal  plan. 

•  The  appraisal  team  is  trained  and  is  familiar  with  the  appraisal  plan. 

Inputs 

•  appraisal  plan 

Activities 

2.1.1  Conduct  Participant  Briefing 

Outputs 

*  prepared  appraisal  participants 

Outcome 

At  the  end  of  this  process,  appraisal  participants  are  prepared  to  provide 
relevant  information  to  the  appraisal  team  and  have  confirmed  their 
participation. 

Exit  Criteria 

A  participant  briefing  has  been  conducted  and  appraisal  participants  are 
prepared  to  participate 

Key  Points 

Inform  members  of  the  organization  who  participate  in  the  appraisal  of 
their  roles,  and  expectations  of  the  sponsor  and  appraisal  team. 

Tools  and 
Techniques 

•  presentation  tools 

•  video  teleconferencing  facilities 

Metrics 

•  planned  and  actual  number  of  participants  briefed 

Verification  and 
Validation 

•  feedback  from  appraisal  participants  on  their  readiness  to  perform 
their  role(s) 

Records 

•  participants  briefed  compared  to  the  plan 

Continued  on  next  page 
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2.1  Prepare  Participants  (continued) 


Tailoring 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


A  variety  of  methods  can  be  used  to  prepare  participants,  including 

•  video  conference  or  teleconference 

•  a  face-to-face  presentation  by  the  appraisal  team  leader  or  designee 

•  multiple  presentations  to  different  audiences  (e.g.,  sponsor, 
interviewees,  and  process  groups) 

•  multiple  presentations  at  different  times  (e.g.,  during  preparation 
activities  and  again  prior  to  the  start  of  interviews) 


This  process  includes  informing  appraisal  participants  of  the  appraisal 
process,  its  purpose  and  objectives,  and  their  roles  in  the  appraisal.  At  a 
minimum,  this  communication  is  addressed  in  a  single  opening  briefing  at 
the  start  of  the  Conduct  Appraisal  phase.  Alternatively,  it  might  be 
addressed  initially  during  the  Plan  and  Prepare  Phase,  but  with  a  short, 
additional  opening  briefing  to  all  participants — again  at  the  start  of  the 
Conduct  Appraisal  phase — to  introduce  the  team  and  reiterate  the  appraisal 
goals.  In  any  event,  it  must  occur  prior  to  the  conduct  of  any  interviews. 


The  activities  in  this  process  serve  to  prepare  participants  for  the  appraisal. 
Participants  are  prepared  prior  to  their  participation  to  ensure  they  are 
aware  of  their  roles  in  the  appraisal,  confirm  their  availability,  and  prepare 
for  their  participation. 


♦ 
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2.1.1  Conduct  Participant  Briefing 

Activity  Members  of  the  organization  who  participate  in  the  appraisal  must  be 

Description  informed  of  their  role  and  the  expectations  the  sponsor  and  appraisal  team 

have.  This  communication  is  typically  accomplished  through  a  briefing  in 
which  the  appraisal  team  leader  provides  an  overview  of  the  appraisal 
process,  purpose,  and  objectives.  Specific  information  about  the  scheduled 
events  and  the  locations  where  they  occur  is  also  communicated  during  this 
presentation,  as  well  as  through  ongoing  contact  between  the 
organizational  unit  coordinator  and  the  members  of  the  organization. 

Required  The  appraisal  team  leader  and/or  designees  shall 

Practices  •  brief  appraisal  participants  on  the  appraisal  process 

•  provide  orientation  to  appraisal  participants  on  their  roles  in  the 
appraisal 

Parameters  Participants  must  reconfirm  their  availability  to  participate  in  the  appraisal. 

and  Limits 

Optional  Provide  orientation  on  the  documentation  of  PIIs  and  any  specific 

Practices  instruments  used,  so  the  appropriate  people  in  the  organization  can 

document  the  initial  objective  evidence  to  be  used  in  the  appraisal. 

Multiple  participant  briefings  may  be  performed  for  separate  parts  of  the 
organization. 

Implementation  The  preparation  of  appraisal  participants  may  be  accomplished  using  a 

Guidance  video  conference  or  teleconference  if  desired. 

Depending  on  the  appraisal  usage  mode  (e.g.,  supplier  selection  versus 
internal  process  improvement),  various  types  of  communication  may  be 
used.  In  the  internal  process  improvement  usage  mode,  the  importance  of 
management  sponsorship  within  the  organization  will  likely  lead  the 
appraisal  team  leader  to  work  with  senior  management  to  help  demonstrate 
commitment  to  the  appraisal  process  as  well  as  the  process  improvement 
work  that  will  follow.  In  the  supplier  selection  usage  mode,  the  possibility 
of  the  same  team  visiting  multiple  organizations  adds  coordination  tasks 
and  communication  channels  as  well. 
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2.2  Examine  Objective  Evidence 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Examine  information  about  the  practices  implemented  in  the  organization 
and  relate  the  resultant  data  to  the  appraisal  reference  model.  Perform  the 
activity  in  accordance  with  the  data  collection  plan.  Take  corrective  actions 
and  revise  the  data  collection  plan  as  needed. 


•  Data  collection  has  been  planned. 

•  The  sponsor  has  approved  the  appraisal  plan. 

•  The  appraisal  team  is  trained  and  is  familiar  with  the  appraisal  plan. 

•  Participants  have  been  informed  about  the  appraisal  process  and  their 
roles  in  it. 


•  Appraisal  data 

-  initial  objective  evidence 

-  documents 

-  documented  practice  implementation  gaps,  if  any 

-  feedback  on  preliminary  findings  (if  that  point  in  the  timeline  has 
been  reached) 

•  Data  collection  plan 

-  appraisal  schedule 

-  interview  schedule 

-  document  list 

-  new  interview  questions 


2.2.1  Examine  Objective  Evidence  from  Documents 

2.2.2  Examine  Objective  Evidence  from  Interviews 


•  updated  appraisal  data 

•  updated  data  collection  plan 


After  the  final  iteration  of  this  process,  the  team  has  sufficient  data  to 
create  appraisal  findings  and  to  make  judgments  about  the  implementation 
of  practices,  as  well  as  the  satisfaction  of  specific  and  generic  goals. 


The  coverage  of  the  appraisal  reference  model  and  the  organizational 
scope  has  been  achieved,  and  the  team  is  ready  to  produce  the  appraisal 
outputs. 


Con  tin  ued  on  next  page 
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2.2  Examine  Objective  Evidence  (continued) 


Key  Points  The  efficient  collection  of  objective  evidence  results  from  carefully 

creating  and  executing  the  data  collection  plan.  Effective  contingency 
planning  and  the  use  of  work  aids  to  monitor  progress  are  key  points  to 
consider.  The  team  must  be  able  to  focus  on  examining  the  most  relevant 
information  available,  rather  than  be  distracted  by  a  mission  to  find  new 
evidence. 

Wall  charts  and  other  visual  aids  are  often  used  to  display  the  results  of 
data  collection  activities.  Electronic  tools  are  prevalent  among  experienced 
appraisal  team  leaders  and  can  be  effective  for  continually  monitoring  and 
updating  the  inventory  of  objective  evidence. 

Metrics  Tracking  the  actual  coverage  obtained,  as  compared  to  the  planned 

coverage,  in  each  data  collection  activity  facilitates  timely  corrective 
actions  where  they  are  needed.  The  most  critical  resource  during  an 
appraisal  is  time.  Using  a  timekeeper  during  data  collection  and 
verification  activities  provides  feedback  on  team  performance.  Recording 
the  actual  duration  of  planned  events  helps  the  team  in  taking  actions  to 
recover  from  unexpected  events. 

Verification  and  The  appraisal  method  provides  detailed  verification  and  validation 

Validation  procedures  for  objective  evidence.  They  are  described  in  process  2.4, 

Verify  Objective  Evidence,  and  2.5,  Validate  Preliminary  Findings. 

Records  Work  aids  used  to  record  and  track  the  progress  of  data  collection  activities 

are  retained  for  traceability  and  provide  an  important  input  to  a  final  report 
describing  the  appraisal,  if  the  sponsor  has  requested  a  final  report.  The 
duration  and  effort  required  for  specific  data  collection  events  can  be 
recorded  to  provide  useful  historical  data  for  planning  subsequent 
appraisals. 

Tailoring  The  method  is  flexible  in  terms  of  the  use  of  customized  data  collection 

instruments,  presentations,  document  reviews,  and  interviews.  Specialized 
forms  of  these  data  collection  methods  can  be  constructed  to  meet  the 
objectives  of  the  appraisal.  Standardized  presentations  can  be  employed  to 
provide  the  team  with  an  “inbrief  ’  at  the  start  of  the  appraisal.  The  method 
also  provides  flexibility  in  choosing  the  number,  duration,  style,  and  make¬ 
up  of  interview  sessions  within  specified  boundaries. 

Continued  on  next  page 
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2.2  Examine  Objective  Evidence  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 
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The  activities  that  provide  the  team  with  the  data  needed  to  produce 
reliable  appraisal  outputs  are  perhaps  the  most  visible  part  of  the  appraisal 
process  from  the  perspective  of  the  appraised  organization.  For  this  reason, 
SCAMPI  A  places  a  heavy  emphasis  on  methodically  planning  and 
tracking  the  data  collected  during  an  appraisal.  The  initial  objective 
evidence  collected  early  in  the  process  allows  team  members  to  analyze  the 
state  of  information  available  at  the  earliest  stages  of  the  appraisal  and 
narrow  their  search  for  new  information.  This  early  work  serves  to 
facilitate  an  efficient  use  of  time.  An  explicit  understanding  of  what 
information  is  needed  and  how  that  information  will  be  used  therefore 
drives  the  activities  associated  with  this  process. 


The  members  of  the  team  continually  manage  the  data  collected  previously 
and  target  new  data  collection  activities  to  fill  known  information  needs. 
Instruments  can  be  used  early  in  the  appraisal  process  and  often  provide 
leads  to  be  pursued  through  other  data  collection  activities  in  addition  to 
affirmations  of  implemented  practices.  Presentations  are  sometimes  used  to 
provide  a  flexible  forum  where  members  of  the  organization  can  explain 
important  information  about  the  practices  implemented  in  the  organization. 
Documents  provide  the  most  explicit  and  lasting  representation  of  practice 
implementation  in  the  organization,  and  the  team  uses  them  to  understand 
how  practices  in  the  reference  model  are  implemented.  Finally,  interviews 
are  used  as  the  most  dynamic  data  collection  technique,  allowing  for 
branching  among  related  topics  and  the  explanation  of  contextual 
information  that  affects  the  implementation  of  practices  as  well  as 
alternative  practices. 

The  appraisal  activities  conducted  for  each  of  these  data  collection  sources 
are  similar: 

•  Determine  if  the  information  obtained  is  acceptable  as  objective 
evidence. 

•  Relate  the  objective  evidence  to  corresponding  practices  in  the 
appraisal  reference  model. 

•  Relate  the  objective  evidence  to  the  appropriate  part  of  the  appraised 
organizational  unit  (i.e.,  the  sample  project  or  support  group). 


♦ 
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2.2.1  Examine  Objective  Evidence  from  Documents 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


CMU/SEI-2006- 


A  substantial  portion  of  the  data  used  by  appraisal  team  members  is 
derived  from  documents  they  review.  Most  of  the  direct  artifacts  used  as 
indicators  of  practice  implementation  are  documents.  Document  review  is 
an  effective  means  to  gain  detailed  insight  about  the  practices  in  use  in  the 
organization.  However,  without  a  clear  focus  on  the  data  being  sought, 
document  review  can  consume  a  great  deal  of  time  as  team  members 
sometimes  attempt  to  read  everything  in  hopes  that  something  useful  will 
be  discovered. 

Objective  evidence  obtained  from  documents  and  from  other  sources  is 
documented  in  process  2.3,  Document  Objective  Evidence,  and  verified  in 
process  2.4,  Verify  Objective  Evidence. 


The  appraisal  team  shall 

•  establish  and  maintain  an  inventory  of  documents  used  as  a  source  of 
objective  evidence 

•  review  information  obtained  from  documents  and  determine  if  it  is 
acceptable  as  objective  evidence 

•  determine  the  model  practices  corresponding  to  the  objective  evidence 
obtained  from  documents 

•  determine  the  portions  of  the  organizational  unit  that  correspond  to 
the  objective  evidence  obtained  from  documents 


Documents  must  be  used  to  determine  the  extent  to  which  practices  have 
been  implemented  in  the  organizational  unit. 

Documents  used  as  objective  evidence  must  have  been  created  or  revised 
prior  to  the  start  of  the  Conduct  Appraisal  phase. 


Con  tin  ued  on  next  page 
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2.2.1  Examine  Objective  Evidence  from  Documents  (continued) 


Optional  For  organizations  with  substantial  intranets  containing  Web-based 

Practices  document  libraries,  a  member  of  the  organization  familiar  with  the 

document  library  should  provide  a  demonstration  of  the  Web-based  tools. 
Links  to  other  documents  and  other  features  of  the  Web-based  document 
library  should  be  tested  prior  to  the  team’s  use  during  the  appraisal. 

Instruments,  such  as  questionnaires  and  objective  evidence  descriptions,  as 
well  as  presentation  materials,  can  provide  documented  objective  evidence. 


Implementation  The  inventory  should  be  sufficient  to  summarize  the  documentation 

Guidance  objective  evidence  used  as  a  basis  for  appraisal  ratings  generated,  as 

required  by  the  appraisal  record  described  in  activity  3.2.2,  Generate 
Appraisal  Record.  Much  of  the  inventory  contents  can  be  obtained  from 
the  mapping  data  or  instruments  obtained  from  the  organizational  unit, 
such  as  the  PII  database  or  questionnaires.  The  inventory  can  be  used  to 
maintain  a  list  of  documents  reviewed  or  additional  documentation 
requested  from  the  organizational  unit. 

One  or  more  team  members  will  seek  data  for  every  practice  in  the 
appraisal  reference  model  scope  of  the  appraisal  through  document  review. 
This  review  does  not  require  a  document  for  every  practice,  as  any  given 
document  is  likely  to  provide  data  relevant  to  multiple  practices.  To  the 
greatest  extent  possible,  the  location  of  documented  evidence  relating  to 
every  practice  should  be  recorded  in  advance  of  the  team’s  arrival  at  the 
site  where  the  appraisal  will  occur.  Organizations  with  established 
improvement  infrastructures  typically  maintain  this  type  of  information  to 
track  their  improvement  efforts  against  the  model.  Where  this  information 
is  incomplete,  the  team  will  be  forced  to  discover  the  links  between  the 
reference  model  and  the  organization’s  implemented  practices,  and  will 
therefore  require  more  time  to  perform  the  appraisal. 


Continued  on  next  page 
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2.2.1  Examine  Objective  Evidence  from  Documents  (continued) 


Implementation 

Guidance 

Three  Levels  of 
Documents 


Documents  reviewed  during  an  appraisal  can  be  classified  into  three 
different  levels:  organization,  project,  and  implementation. 

By  providing  further  insight  into  the  policies  and  procedures  that  guide  the 
organization’s  processes,  organization- level  documents  sometimes  help  the 
team  to  eliminate  the  need  for  a  question  during  an  interview  or  sharpen 
the  focus  for  a  question.  Review  of  these  documents  provides  a  context  for 
understanding  the  expectations  placed  on  projects  and  support  groups 
within  the  organization. 

Through  the  review  of  instantiation-level  documents,  team  members  gain 
further  insight  into  each  scheduled  interviewee’s  role  in  their  respective 
project  or  support  group  they  support  as  well  as  the  terminology  generally 
accepted  within  the  organization,  project,  or  support  group.  This  review 
may  lead  to  the  refinement  or  modification  of  interview  questions. 

The  team  typically  reviews  implementation-level  documents  to  validate 
information  gathered  from  other  sources,  such  as  interviews  or  higher  level 
documents.  Documents  on  this  level  provide  an  audit  trail  of  the  processes 
used  and  the  work  performed.  The  review  of  these  documents  frequently 
provides  verification  of  practices  found  in  organization-  and  instantiation- 
level  documents. 
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2.2.2  Examine  Objective  Evidence  from  Interviews 


Activity 

Description 


Required 

Practices 
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Interviews  are  used  to  obtain  oral  affirmations  related  to  the 
implementation  of  processes  within  the  organizational  scope  of  the 
appraisal.  Interviews  are  held  with  managers  and  practitioners  responsible 
for  the  work  being  performed.  The  appraisal  team  uses  interviews  to 
understand  how  the  processes  are  implemented  and  to  probe  areas  where 
additional  coverage  of  model  practices  is  needed. 

Interviews  are  a  required  and  necessary  component  of  a  SCAMPI  A 
appraisal  in  all  usage  modes.  The  criteria  for  the  amount  of  oral  affirmation 
objective  evidence  to  be  collected  are  described  in  activity  2.4.1,  Verify 
Objective  Evidence.  These  criteria  drive  the  development  of  the  initial 
interviewing  strategy  in  the  data  collection  plan  described  in  activity  1.5.2, 
Prepare  Data  Collection  Plan.  A  variety  of  interviewing  techniques  are 
available,  and  the  appraisal  team  leader  works  with  the  team  to  schedule 
the  most  appropriate  interview  types  for  the  situation. 

As  objective  evidence  is  gathered  throughout  the  appraisal,  the  data 
collection  plan  is  revised  as  necessary.  By  using  focused  investigation 
techniques,  the  need  for  interviews  may  be  either  increased  or  diminished, 
as  long  as  the  criteria  for  oral  affirmations  are  satisfied. 

Objective  evidence  obtained  from  interviews  and  other  sources  is 
documented  in  process  2.3,  Document  Objective  Evidence,  and  verified  in 
process  2.4,  Verify  Objective  Evidence. 


The  appraisal  team  shall 

•  refine  the  data  collection  plan  to  determine  the  objective  evidence  that 
must  be  obtained  from  interview  participants 

•  conduct  interviews  to  obtain  information  that  may  be  used  as 
objective  evidence 

•  review  information  obtained  from  interviews  and  determine  if  it  is 
acceptable  as  objective  evidence 

•  determine  the  model  practices  corresponding  to  the  objective  evidence 
obtained  from  interviews 

•  determine  the  portions  of  the  organizational  unit  that  correspond  to 
the  objective  evidence  obtained  from  interviews 


Con  tin  ued  on  next  page 
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2.2.2  Examine  Objective  Evidence  from  Interviews  (continued) 


Parameters 
and  Limits 


Optional 

Practices 


All  SCAMPI  A  appraisals  must  use  interviews  as  a  source  of  information 
on  the  extent  to  which  practices  have  been  implemented  in  the 
organizational  unit  and  within  the  sampled  projects  and  support  groups. 

All  interviews  must  include  at  least  two  members  of  the  appraisal  team 
designated  by  the  appraisal  team  leader. 

Whenever  virtual  methods  such  as  video  conferences,  teleconferences,  and 
other  similar  technologies  are  used  for  interviews,  the  appraisal  team 
leader  must  ensure  that  these  methods  do  not  compromise  the  integrity  or 
accuracy  of  the  appraisal  activities  or  the  appraisal  results. 

Full  coverage  of  the  reference  model,  the  organizational  scope,  and  the 
organization’s  lifecycle(s)  in  the  appraisal  scope  must  be  achieved  with 
the  objective  evidence  considered  by  the  team.  Therefore,  the  pool  of 
potential  interviewees  must  cover  all  elements  of  the  process  in  use  in  the 
organizational  unit. 

Steps  must  be  taken  to  ensure  open  communication  during  interviews  by 
addressing  potential  issues  among  interviewees  and  team  members  (e.g., 
presence  of  supervisors  or  process  owners). 

The  rules  of  confidentiality  and  the  expected  use  of  appraisal  data  must  be 
communicated  to  every  interviewee. 


Request  that  interviewees  bring  a  document  or  other  artifact  with  them  to 
their  interviews  for  a  “show-and-tell”  style  interview. 

Presentations  that  include  appraisal  team/organization  interaction  may  be 
used  as  interviews. 

Use  video  conference,  teleconference,  or  other  similar  technology  to 
conduct  interviews  at  a  distance. 


Con  tin  ued  on  next  page 
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2.2.2  Examine  Objective  Evidence  from  Interviews  (continued) 


Implementation 

Guidance 


Project  and/or  organization  management  personnel  are  typically 
interviewed  individually,  or  grouped  according  to  project  or  support  group. 
The  focus  of  the  discussion  in  these  interviews  will  therefore  be  scoped  to 
a  particular  project  or  support  group,  rather  than  across  the  sampled 
projects  or  support  groups. 

Functional  area  representatives  (FARs)  are  typically  interviewed  in  a  group 
across  the  projects  and  support  groups  within  the  organizational  scope  of 
the  appraisal.  The  focus  of  the  discussion  in  these  interviews  will  therefore 
be  scoped  to  a  particular  set  of  practices  used  across  the  instantiations. 

Interviews  provide  the  most  flexible  source  of  detailed  data.  Oral 
interaction  with  people  who  enact  the  practices  being  investigated  allows 
the  team  to  seek  detailed  information  and  to  understand  the 
interrelationships  among  various  practices.  Detailed  information  to  address 
specific  data  collection  needs  can  be  sought  and  verified  in  real  time. 

It  is  important  to  avoid  sampling  interviewees  for  a  session  such  that  two 
people  in  the  same  reporting  chain  (e.g.,  a  superior  and  one  of  his/her 
direct  reports)  are  in  the  same  interview  session.  This  restriction  applies  to 
members  of  the  appraisal  team  as  well.  People  who  have  this  type  of 
relationship  with  one  another  may  be  uncomfortable  with  the  expectation 
for  them  to  be  completely  candid  during  the  interview. 

Samples  of  interviewees  are  typically  grouped  into  categories  that  roughly 
correspond  to  lifecycle  phases,  engineering  disciplines,  organizational 
groupings,  and/or  PA  affinities.  As  stated  previously,  interviews  of 
management  personnel  are  typically  grouped  by  project  or  support  group, 
while  FARs  sampled  for  a  given  interview  come  from  across  the 
organizational  unit. 

Virtual  methods  should  allow  for  adequate  interaction  between  the 
appraisal  team  members  and  the  appraisal  participants  and  should  provide 
mechanisms  for  the  appraisal  team  to  control  the  interviews  (e.g.,  provide 
the  ability  to  interrupt,  ask  questions,  or  redirect  the  discussion  to  other 
subjects). 

There  are  three  basic  forms  of  interviews  used  in  SCAMPI  A.  They  are 
described  below. 


Con  tin  ued  on  next  page 
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2.2.2  Examine  Objective  Evidence  from  Interviews  (continued) 


Implementation  The  most  structured  approach  is  the  standard  interview,  which  is  scheduled 

Guidance  in  advance  and  employs  a  series  of  scripted  questions.  Each  standard 

interview  typically  involves  interviewees  with  similar  responsibilities  in 
Standard  the  organization  (e.g.,  quality  assurance  personnel,  systems  engineers,  or 

Interviews  middle  managers).  The  schedule  and  location  of  each  interview  session  is 

communicated  to  the  interviewees  well  in  advance.  Questions  intended  to 
elicit  data  about  particular  practices  are  prepared  and  reviewed  in  advance, 
and  the  team  follows  a  defined  process  for  conducting  the  session.  Often, 
the  entire  team  is  present  for  these  interviews.  Responsibility  for  tracking 
the  coverage  of  individual  PAs  is  typically  assigned  to  team  members.  A 
single  questioner  may  lead  the  interview,  with  the  rest  of  the  team  listening 
and  taking  notes,  or  the  responsibility  for  asking  questions  may  be 
distributed  among  the  team  members.  In  any  case,  it  is  expected  that  all 
team  members  who  are  not  asking  questions  listen  and  take  notes  for  all 
questions. 

A  set  of  planned  interviews  will  be  defined  during  appraisal  planning.  As 
the  appraisal  progresses  and  the  objective  evidence  accumulates,  the  team 
may  find  it  convenient  to  cancel  one  or  more  of  these  interviews  to  use  the 
time  for  other  activities.  Such  changes  in  the  data  collection  plan  are  made 
in  a  way  that  does  not  violate  the  coverage  criteria  described  in  process 
2.4,  Verify  Objective  Evidence. 


Implementation  A  more  flexible  approach  to  scheduling  interviews  is  available  in  the  form 
Guidance  of  on-call  interviews,  a  variant  of  the  standard  interview.  Prospective 

interviewees  are  identified  and  notified  in  advance,  just  as  described 
On-Call  above.  However,  the  interviews  are  only  held  if  team  members  decide  that 

Interviews  there  is  a  need  and  that  the  time  will  be  well  spent.  The  prospective 

interviewees  are  therefore  asked  to  block  a  period  of  time  for  such  a 
contingency,  and  are  informed  the  day  before  the  scheduled  time  as  to 
whether  or  not  the  interview  session  will  actually  happen.  These  interviews 
need  not  include  the  entire  appraisal  team,  thus  permitting  parallel  sessions 
with  different  interviewees. 


Con  tin  ued  on  next  page 
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2.2.2  Examine  Objective  Evidence  from  Interviews  (continued) 


Implementation 

Guidance 

Office  Hours 
Interviews 


Finally,  office  hours  interviews  represent  an  agreement  for  availability  that 
permits  pairs  of  team  members  to  visit  interviewees  at  their  desks, 
cubicles,  or  offices.  As  with  the  on-call  interviews,  the  prospective 
interviewees  block  a  specific  time  period  to  be  available  on  a  contingency 
basis.  Most  prospective  interviewees  will  be  able  to  continue  with  their 
daily  work  and  accommodate  an  interruption  if  the  team  needs  to  speak 
with  them.  Here  again,  only  if  specific  data  needs  are  identified  will  the 
interview  occur.  The  interviewees  should  be  informed  that  they  may 
receive  only  limited  advanced  notice  for  these  interviews,  although 
confirming  the  interview  at  least  a  day  in  advance  is  a  courtesy  that  should 
be  offered  whenever  possible. 
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2.3  Document  Objective  Evidence 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Create  lasting  records  of  the  information  gathered  by  identifying  and  then 
consolidating  notes,  transforming  the  data  into  records  that  document 
practice  implementation  as  well  as  strengths  and  weaknesses. 


Planning  activities  for  the  appraisal  are  complete,  including  the  selection 
and  preparation  of  the  appraisal  team.  At  least  one  data  collection  activity 
has  been  conducted,  and  appraisal-relevant  data  are  available  to  record. 


•  appraisal  data 

•  notes  taken  during  data  collection  activities  (if  applicable) 

•  annotated  worksheets  or  other  work  aids  containing  data  (if 
applicable) 

•  strengths  and  weaknesses  documented  from  previous  activities 

•  data  collection  plan 


2 . 3 . 1 .  T ake/Review/T ag  N otes 

2.3.2.  Record  Presence/Absence  of  Objective  Evidence 

2.3.3.  Document  Practice  Implementation 

2.3.4.  Review  and  Update  the  Data  Collection  Plan 


•  updated  appraisal  data 

-  tagged  notes 

-  noted  practice  implementation  gaps  (if  any) 

-  revised  data  collection  plan  (if  applicable) 

-  annotated  worksheets 

•  requests  for  additional  data  (interviewees  or  documents) 


Individual  team  members  understand  the  data  collected  thus  far,  and  have 
information  to  guide  any  needed  subsequent  data  collection. 


All  data  from  the  most  recent  data  collection  session  has  been  captured  as  a 
new  baseline  of  practice  implementation  evidence  or  strength  and 
weakness  statements.  The  data-gathering  plans  have  been  updated  to 
reflect  additional  information  needed  and  topics  that  can  be  removed  from 
active  investigation. 


Con  tin  ued  on  next  page 
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2.3  Document  Objective  Evidence  (continued) 


Key  Points  This  process  has  traditionally  been  the  most  difficult  one  to  manage  during 

an  appraisal.  Members  of  the  team  will  tend  to  vary  a  great  deal  in  their 
productivity  and  style  of  work.  The  appraisal  team  leader  must  be  attentive 
to  the  progress  of  each  team  member  and  take  effective  corrective  actions 
to  ensure  team  progress. 

Tools  and  Because  of  the  challenging  nature  of  this  activity,  appraisal  team  leaders 

Techniques  tend  to  have  strong  preferences  for  using  tools  and  techniques  they  have 

found  to  be  successful.  Only  a  high-level  list  of  tools  and  techniques  is 
provided  here: 

•  Work  aids  like  wall  charts,  spreadsheet  programs,  and  automated 
database  tools  are  frequently  used  to  help  track  the  status  of  data 
collection. 

•  Using  mini-teams,  where  pairs  (or  triplets)  of  team  members  are 
assigned  specific  PAs  or  specific  projects,  is  a  common  practice. 

•  Time  management  is  a  critical  skill  for  this  activity.  Explicitly 
reviewing  the  effort  spent,  in  real  time,  is  a  useful  way  to  focus  the 
team. 

•  A  variety  of  techniques  for  structuring  team  notebooks  and  formats 
for  recording  notes  has  been  used. 

•  Team  norms  regarding  techniques  for  managing  debates  and  divergent 
views  are  important,  and  should  be  made  explicit  well  in  advance. 

Metrics  As  mentioned  above,  tracking  the  effort  expended  during  this  activity  (in 

real  time)  is  a  valuable  technique  to  manage  the  team’s  time.  The  ability  to 
quickly  learn  the  rate  at  which  each  team  member  works  is  a  skill  that 
experienced  appraisal  team  leaders  develop  using  effort  and  duration 
metrics. 

The  method  rules  for  recording  traceability  and  validating  data  provide 
verification  and  validation  of  the  appraisal  data.  Monitoring  progress  and 
the  consensus  decision-making  process,  conducted  by  the  appraisal  team 
leader,  also  serves  as  important  verification  and  validation  activity. 

Records  All  appraisal  data  are  recorded  with  full  traceability  to  information  sources 

as  well  as  the  model  components  to  which  they  pertain.  The  full  detail  in 
this  traceability  contains  sensitive  information  that  should  not  be  provided 
to  people  outside  of  the  appraisal  team.  The  attribution  of  data  to 
individuals  must  never  be  divulged  even  when  detailed  data  are  provided 
to  the  organization  for  use  in  process  improvement. 

Tailoring  The  use  of  a  specialized  appraisal  data  management  tool  is  common 

tailoring  applied  to  this  activity. 


Continued  on  next  page 
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2.3  Document  Objective  Evidence  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  mechanics  associated  with  the  recording  and  transcription  of  objective 
evidence  are  described  in  this  section.  There  are  many  links  between  these 
mechanics  and  the  data  collection  process,  as  well  as  the  data  verification 
and  validation  process.  The  data-recording  process  must  support  these 
other  processes,  and  the  tools  used  during  an  appraisal  must  accommodate 
these  linkages.  Typically,  an  integrated  database  tool  is  used  to  manage  all 
appraisal  data  that  results  from  the  analysis  of  notes  taken  during  data 
collection. 


The  most  basic  representation  of  appraisal  data  is  found  in  the  notes  taken 
by  individual  team  members.  These  notes  are  reviewed  and  are  typically 
“tagged”  or  otherwise  processed  before  their  content  is  transformed  into 
other  lasting  representations.  The  presence,  absence,  and/or 
appropriateness  of  objective  evidence  is  then  judged  and  recorded  based  on 
the  data  collected.  The  scheme  by  which  this  set  of  records  is  produced  is 
an  important  implementation  choice  made  by  the  appraisal  team  leader, 
and  must  be  well  understood  by  the  team.  Gaps  in  the  implemented 
practices  are  also  recorded  in  a  consistent  manner  that  ensures  traceability. 
Finally,  the  data  collection  plan  is  reviewed  in  light  of  the  changes  in  the 
set  of  data  available  to  the  team  and  the  remaining  data  needed  to  support 
reliable  rating  judgments. 
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2.3.1  Take/Review/Tag  Notes 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


As  team  members  examine  data  sources,  they  document  what  the  objective 
evidence  is  (e.g.,  referencing  documents,  presentations,  instruments,  and 
interviewee  comments),  as  well  as  why  or  how  the  objective  evidence 
meets  the  intent  of  the  model  practice. 

There  may  be  special  cases  where  team  members  elect  to  record  data 
directly  in  the  objective  evidence  tracking  tool.  In  such  cases,  team 
members  may  choose  not  to  take  notes  (on  paper  or  in  their  notebooks)  that 
describe  the  objective  evidence. 

For  all  interviews  and  presentations,  the  team  members  must  take  notes 
that  capture  the  objective  evidence  before  they  move  to  the  annotation  of 
the  objective  evidence  tracking  tool. 


The  appraisal  team  shall 

•  record  notes  obtained  from  objective  evidence  data-gathering  sessions 

•  relate  notes  to  corresponding  practices  in  the  appraisal  reference 
model 


Every  team  member  present  must  take  notes  during  interviews  and 
presentations.  These  notes  must  cover  all  areas  investigated  during  the 
interview,  and  are  not  limited  to  the  PAs  assigned  to  the  individual  team 
member  (i.e.,  everybody  takes  notes  on  everything). 

During  document  reviews,  notes  must  be  taken  to  preserve  specific  context 
or  focused  references,  if  the  rationale  for  accepting  the  objective  evidence 
is  not  self-evident. 


Tagging  schemes  (that  show  traceability  to  model  practices)  and 
techniques  for  highlighting  phrases  are  determined  by  the  preferences  of 
the  note  taker.  A  variety  of  formats  for  team  member  notebooks  has  been 
devised  to  facilitate  note  taking  and  tracking  raw  data  during  appraisals. 
Frequently,  the  questions  used  during  an  interview  will  be  printed  and 
collated  within  a  team  member  notebook  that  contains  note-taking  forms 
and  other  useful  information  like  interview  schedules  and  document  lists. 

Notes  can  be  recorded  for  items  that  have  significant  positive  or  negative 
impact  on  the  enactment  of  processes  within  the  organizational  unit,  even 
if  they  are  not  directly  related  to  model  practices.  These  items  may 
ultimately  be  reflected  in  non-model  findings  reported  to  the  organizational 
unit. 


Con  tinued  on  next  page 
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2.3.1  Take/Review/Tag  Notes  (continued) 


Implementation 

Guidance 


Implementation 

Guidance 

Taking  Notes 


Implementation 

Guidance 

Reviewing 

Notes 


Implementation 

Guidance 

Tagging  Notes 


The  raw  notes  taken  during  an  appraisal  are  treated  as  confidential 
information  and  may  not  be  provided  to  any  person  outside  of  the  appraisal 
team.  Team  members  are  typically  required  to  destroy  their  notes  in  a 
secure  manner  at  the  conclusion  of  the  appraisal.  This  requirement  ensures 
that  the  attribution  of  detailed  information  to  individuals  in  the 
organization  cannot  lead  to  inappropriate  consequences  following  the 
appraisal. 


Team  members  actively  take  notes  during  all  data-gathering  sessions.  The 
purpose  is  to  record,  verbatim,  what  the  information  source  reveals  about 
the  implementation  of  practices  in  the  project  or  organization. 

Whenever  notes  are  taken  in  a  data-gathering  session,  individual  team 
members  should  review  their  notes  immediately  after  the  conclusion  of  the 
session.  The  review  will  focus  on  tagging  significant  items  that  relate  to 
one  or  more  model  practice(s).  This  review  and  tagging  process  should 
occur  within  24  hours  of  the  data-gathering  session. 


The  context  in  which  the  data  are  provided — be  it  during  an  interview, 
presentation,  or  in  a  document — affects  the  proper  interpretation  of  the 
facts.  For  example,  notes  taken  during  an  interview  are  based  on  a  give  and 
take  between  the  interviewer  and  the  interviewee.  The  threads  of 
discussion  often  provide  a  context  that  may  not  be  reflected  in  a  single 
exceipt  from  the  middle  of  the  interview.  Note-takers  should  review  their 
work  to  ensure  that  such  contextual  information  can  be  preserved  at  least 
in  their  recollection,  and  preferably  through  the  annotation  of  the  notes. 


As  notes  are  reviewed,  team  members  often  use  highlighter  pens  or 
annotation  schemes  to  identify  the  most  salient  excerpts.  The  PA  and/or 
practice  to  which  the  information  applies  may  be  written  in  colored  ink 
over  the  raw  notes.  All  notes  should  identify  the  data-gathering  session, 
and  the  pages  should  be  numbered  to  preserve  the  sequence  of  information. 
For  notes  taken  during  interviews,  it  is  often  useful  to  draw  a  seating  chart 
to  show  where  each  person  was  sitting  during  the  interview.  Scripts 
prepared  in  advance  of  scheduled  interviews  may  already  be  tagged,  and 
can  help  relate  responses  to  appropriate  sections  of  the  appraisal  reference 
model.  Some  interviewee  responses  may  deal  with  model  practices  other 
than  those  targeted  by  a  given  question,  which  would  still  necessitate  some 
additional  tagging. 
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2.3.2  Record  Presence/Absence  of  Objective  Evidence 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Page  11-84 


The  presence  or  absence  of  appropriate  objective  evidence  for  each  model 
practice  in  the  scope  of  the  appraisal  is  determined  based  on  information 
obtained  from  data-gathering  sessions.  Annotations  are  recorded  indicating 
the  source,  relevance,  and  coverage  of  objective  evidence  collected.  In 
situations  where  just  referencing  the  data  source  would  not  make  it  obvious 
why  the  objective  evidence  is  appropriate,  a  comment  can  be  added  to  the 
annotation.  For  example,  when  an  alternative  to  the  typical  work 
breakdown  structure  is  used,  it  may  be  necessary  to  document  why  that 
alternative  meets  the  intent  of  the  model  practice.  Adding  comments  to  the 
annotations  can  help  to  avoid  rehashing  the  rationale  for  accepting  the 
objective  evidence  multiple  times  during  team  discussions. 


The  appraisal  team  shall 

•  for  each  instantiation,  record  the  presence  or  absence  of  objective 
evidence  collected  for  each  reference  model  practice  within  the 
appraisal  scope 


The  inventory  of  objective  evidence  (be  it  in  electronic  or  paper  form)  is 
updated  to  reflect  what  the  data  imply  about  the  implementation  of 
particular  practices.  For  every  practice  within  the  reference  model  scope  of 
the  appraisal,  annotations  indicating  the  presence  or  absence  of  objective 
evidence  will  be  made  throughout  the  appraisal  conduct.  The  annotation 
scheme  used  must  ensure  that  the  record  reveals  the  following  information: 

•  the  project  or  support  group  to  which  the  data  apply 

•  the  specific  or  generic  practice  to  which  the  data  apply 

•  the  type  of  objective  evidence  being  recorded  (i.e.,  direct,  indirect,  or 
affirmation) 

•  whether  the  data  imply  the  presence  or  absence  of  the  objective 
evidence 

•  whether  the  data  suggest  that  the  objective  evidence  is  appropriate 

•  comments  about  the  appropriateness  of  the  evidence  (if  needed) 

•  whether  or  not  additional  information  is  needed  before  the  team  can 
characterize  the  extent  to  which  the  practice  is  implemented 

•  a  description  of  what  the  evidence  is,  if  such  a  description  was  not 
provided  by  the  organization  in  advance 


Con  tin  ued  on  next  page 
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2.3.2  Record  Presence/Absence  of  Objective  Evidence  (continued) 


Optional  Following  each  verification  session  where  the  presence  or  absence  of 

Practices  objective  evidence  is  recorded,  the  team  reviews  the  judgments  about  each 

new  piece  of  objective  evidence.  This  review  may  be  useful  in  establishing 
a  common  understanding  of  the  expectations  for  objective  evidence, 
especially  early  in  the  appraisal. 

Implementation  This  activity  represents  the  mechanical  aspects  of  processing  appraisal 
Guidance  data,  and  is  strongly  tied  to  the  activities  described  in  process  2.4,  Verify 

Objective  Evidence.  The  emphasis  of  this  activity  description  is  on  the 
steps  needed  to  update  the  inventory  of  objective  evidence  and  maintain 
traceability  to  data  sources.  The  emphasis  of  the  activity  description  in 
Verify  Objective  Evidence  is  on  the  interpretation  of  data  collected  and  the 
sufficiency  of  objective  evidence  relative  to  the  appraisal  reference  model. 

Team  members  typically  record  the  presence  or  absence  of  appropriate 
objective  evidence  into  tools  such  as  tracking  tables  or  data  consolidation 
worksheets.  Prior  to  the  assignment  of  goal  ratings,  the  entire  team  reviews 
the  status  of  the  objective  evidence  as  reflected  in  the  annotations  made  by 
each  team  member. 

The  data  gathered  during  every  data  collection  session  should  be  related  to 
the  practices  in  use  in  a  project  or  support  group  within  the  organizational 
unit.  In  recording  the  presence  or  absence  of  objective  evidence,  the  intent 
is  to  quickly  inventory  the  composite  of  factual  information.  Elaboration 
about  what  the  data  mean  or  how  they  relate  to  other  important  issues  is 
captured  either  in  notes  or  in  the  descriptions  of  practice  implementation 
gaps  crafted  by  team  members. 
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2.3.3  Document  Practice  Implementation 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  primary  intent  of  this  activity  is  to  derive,  from  the  objective  evidence 
gathered,  summary  statements  that  describe  the  gap  between  what  the 
objective  evidence  shows  and  what  the  team  was  looking  for  to  support  a 
claim  that  the  model  practice  was  implemented.  The  statements  explain 
why  the  practice  is  not  considered  to  be  Fully  Implemented.  Preliminary 
findings,  including  statements  of  practice  implementation  gaps  will  be 
validated  with  the  appraisal  participants  at  a  later  time. 

Strengths  are  not  recorded  pro  forma  when  practices  are  found  to  be  Fully 
Implemented.  Where  practices  represent  exemplary  implementations  of  the 
model  practices,  the  appraisal  team  will  highlight  these  practices  as  part  of 
the  appraisal  output.  However,  the  primary  focus  of  this  benchmarking 
method  is  to  help  the  organization  verify  the  implementation  of  the  model 
and  identify  areas  where  work  is  needed. 


The  appraisal  team  shall 

•  document  gaps  in  the  project  or  support  group’s  implemented 
processes  relative  to  appraisal  reference  model  practices 


For  any  practice  that  is  characterized  as  something  other  than  Fully 
Implemented,  there  must  be  a  statement  explaining  the  gap  between  what 
the  organization  does  and  what  the  model  expects. 

Statements  of  practice  implementation  gaps,  presented  to  the 
organizational  unit  in  the  form  of  preliminary  findings  for  validation,  must 
be  free  of  references  to  specific  individuals  or  projects,  unless  project-level 
findings  are  planned  for  the  appraisal. 

Regardless  of  the  medium  used,  statements  describing  practice 
implementation  gaps  must  be  annotated  with  the  following  identifying 
information: 

•  the  model  component  to  which  the  statement  relates  (i.e.,  PA,  goal, 
and  practice) 

•  the  data  collection  session(s)  in  which  the  information  was  uncovered 

•  the  project  or  support  group  to  which  the  statement  applies 


Con  tin  ued  on  next  page 
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2.3.3  Document  Practice  Implementation  (continued) 


Optional  Document  strengths  in  the  implementation  of  model  practices  when  the 

Practices  team  discovers  exemplary  implementations. 

Label  implementation  gaps  as  “opportunities  for  improvement”  to  avoid 
the  potentially  negative  connotations  of  labeling  them  as  weaknesses. 

Document  any  significant  issues  impeding  performance  in  the  organization 
that  do  not  necessarily  map  to  the  reference  model.  Identifying  these  issues 
should  be  done  cautiously,  and  the  number  of  these  issues  should  not  be 
larger  than  the  number  of  model-related  issues  reported  by  the  team. 


Implementation  The  database  used  to  record  the  inventory  of  objective  evidence  may 
Guidance  incorporate  functionality  to  record  practice  implementation  gaps  and 

strengths,  or  a  separate  location  or  tool  may  be  used  if  desired.  Gaps  in 
practice  implementation  should  be  recorded  at  the  level  of  a  particular 
instance  of  a  model  practice.  These  precursors  to  preliminary  findings  are 
more  detailed  and  pointed,  while  all  information  presented  outside  of  the 
team  will  be  aggregated  to  the  goal  and  organizational  unit  level  of 
abstraction. 

Strengths  are  only  documented  if  the  implementation  of  a  practice  is 
exceptional,  and  reflects  a  strong  asset  in  the  process  in  use.  An  adequate 
implementation  of  a  model  practice  is  not  necessarily  a  strength.  Team 
members  should  use  their  collective  experience  and  judgment  to  determine 
whether  or  not  they  have  uncovered  an  exemplary  practice  (above  and 
beyond  the  capability  described  in  the  model)  to  highlight  in  the  appraisal 
output. 

Gaps  in  practice  implementation  are  documented  if  the  objective  evidence 
indicates  a  missing  component  in  the  process  or  an  inappropriate  practice, 
in  light  of  the  value  the  practice  is  expected  to  add  to  the  achievement  of 
the  goal.  That  is,  practices  that  fail  to  help  the  organization  meet  the 
appraisal  reference  model  goal  to  which  they  relate  should  have  a  gap 
documented  that  explains  why  the  goal  is  not  met. 
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2.3.4  Review  and  Update  the  Data  Collection  Plan 

Activity 

Description 

This  activity  is  used  to  continuously  monitor  the  state  of  available 
objective  evidence  and  to  select  the  next  tactic  in  the  pursuit  of  obtaining 
full  coverage  of  the  reference  model  scope  and  organizational  scope  of  the 
appraisal. 

Required 

Practices 

The  appraisal  team  shall 

•  review  the  inventory  of  objective  evidence  collected  and  the  data 
collection  plan  to  determine  what  additional  objective  evidence  is  still 
needed  for  sufficient  coverage  of  the  appraisal  reference  model  scope 

•  revise  the  data  collection  plan  to  obtain  additional  objective  evidence 
for  instances  where  insufficient  data  are  available  to  judge  the 
implementation  of  appraisal  reference  model  practices 

•  identify  priorities  for  the  upcoming  data  collection  events  and 
reevaluate  the  feasibility  of  the  schedule  in  light  of  the  current  state  of 
the  objective  evidence 

Parameters 
and  Limits 

This  activity  must  be  enacted  at  least  once  a  day,  and  a  consolidated 
summary  of  the  appraisal  data  collection  status  must  be  available  to  the 
team  at  the  start  of  each  day  during  which  data  collection  events  are 
planned. 

Optional 

Practices 

In  addition  to  the  daily  status  mentioned  above,  more  frequent  status 
checks  may  be  conducted.  These  interim  status  checks  are  not  aggregated 
across  the  team  for  a  team-wide  view  of  status,  unless  the  appraisal  team 
leader  finds  that  beneficial. 

Implementation 

Guidance 

The  data  collection  status  summarizes  the  differences  between  the 
objective  evidence  on  hand  and  the  evidence  needed  to  support  the  creation 
of  appraisal  outputs  (e.g.,  ratings).  Annotations  regarding  the  presence  (and 
appropriateness)  of  objective  evidence  allow  the  team  to  inventory  the 
state  of  the  “knowledge  base.”  This  status  then  drives  requirements  for  the 
collection  of  more  data.  The  annotation  of  the  inventory  of  objective 
evidence  is  described  in  process  2.4,  Verify  Objective  Evidence. 

The  plan  for  future  data  collection  should  be  revisited  and  updated  as 
necessary.  There  may  be  several  situations  in  which  additional  data  are 
required  for  the  team  to  sufficiently  characterize  the  implementation  of 
appraisal  reference  model  practices.  The  following  are  examples  of  such 
situations: 

•  The  process  of  reconciling  new  data  with  the  old  may  identify 
conflicts  or  ambiguities  in  the  data  that  require  clarification. 

•  The  search  for  objective  evidence  may  lead  to  the  discovery  of  one  or 
more  previously  undocumented  practice(s)  in  the  organization. 

•  Attempts  to  confirm  the  use  of  a  particular  practice  or  tool  by  a 
project  or  support  group  may  have  been  unsuccessful. 

Continued  on  next  page 
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2.3.4  Review  and  Update  the  Data  Collection  Plan  (continued) 


Implementation 

Guidance 

(continued) 


Specific  information  needed  to  resolve  ambiguities  or  conflicts  in  the 
existing  data  should  be  documented  for  follow-up  by  one  or  more  members 
of  the  team.  For  detailed  data  items  that  have  a  limited  scope  of  impact,  the 
notes  of  individual  team  members  may  be  adequate  to  document  the  data 
needed.  For  example,  whether  or  not  a  particular  person  is  involved  in  a 
meeting,  or  reviews  a  given  document,  can  be  confirmed  by  a  simple 
question  asked  during  an  on-call  interview.  Therefore,  a  note  made  by  an 
individual  team  member  to  make  sure  the  question  is  asked  may  suffice.  In 
contrast,  if  conflicting  information  is  uncovered  about  whether  or  not  a 
given  event  occurred  (e.g.,  a  meeting)  more  visibility  of  this  conflict  may 
be  needed  among  the  team  members  to  understand  why  the  information 
collected  thus  far  is  not  internally  consistent.  In  such  a  case,  the  person(s) 
responsible  for  the  PA  in  which  that  practice  resides  may  need  to  alert  the 
team  to  the  conflicting  data  and  facilitate  a  team  discussion  to  seek  clarity, 
as  well  as  additional  data.  This  potential  conflict  may  lead  to  the  crafting 
of  a  specific  interview  question,  which  is  used  in  a  standard  interview. 

The  data  collection  plan  and  inventory  of  objective  evidence  provide  a 
means  for  the  appraisal  team  to  continuously  monitor  progress  toward 
sufficient  coverage  of  appraisal  reference  model  practices  in  preparation 
for  rating.  Estimates  of  the  additional  data  collection  effort  should  be 
regularly  reviewed.  If  the  feasibility  of  the  appraisal  schedule  is  called  into 
question,  a  replanning  effort  may  be  necessary  (as  described  in  activity 
1.5.3,  Replan  Data  Collection). 


Prioritizing  data  needs  and  allocating  data  collection  effort  to  particular 
data  collection  events  are  ongoing  activities  that  the  appraisal  team  leader 
is  responsible  for  overseeing.  The  data  collection  status  summary  may  be 
maintained  by  the  appraisal  team  leader  and  reported  to  the  team  members, 
or  the  appraisal  team  leader  may  elect  to  have  each  mini-team  perform  this 
activity  for  the  PAs  it  is  assigned. 
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2.4  Verify  Objective  Evidence 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Verify  the  implementation  of  the  organization’s  practices  for  each 
instantiation,  describing  gaps  in  the  implementation  of  model  practices. 
Each  implementation  of  each  practice  is  verified  so  that  it  may  be 
compared  to  the  practices  of  the  reference  model.  Then  the  team 
characterizes  the  extent  to  which  the  practices  in  the  model  are 
implemented.  Exemplary  implementations  of  model  practices  may  be 
highlighted  as  strengths  to  be  included  in  appraisal  outputs. 


Objective  evidence  has  been  collected  about  the  implementation  of 
practices  in  the  organization.  Gaps  in  the  implementation  of  model 
practices  have  been  identified,  and  the  team  is  ready  to  characterize  the 
extent  to  which  model  practices  (or  acceptable  alternatives  to  those 
practices)  have  been  implemented. 


•  appraisal  plan,  including  schedule  and  participants  for  verification 
activities 

•  data  on  practice  implementation  and  strength/weakness  statements 

•  data  collection  plan  specifying  any  additional  information  needed 


2.4. 1  Verify  Objective  Evidence 

2.4.2  Characterize  Implementation  of  Model  Practices 


•  updated  appraisal  data 

-  strength/weakness  statements 

-  annotated  worksheets 

•  updated  appraisal  artifacts 

-  preliminary  findings 

-  revised  data  collection  plan 

-  requests  for  additional  data 

•  practice  characterizations 

-  instantiation  level 

-  organizational  unit  level 


The  team’s  confidence  in  the  material  that  will  form  the  basis  for  appraisal 
outputs  is  increased.  Any  critical  deficiencies  in  the  data  on  hand  have 
been  identified  and  actions  to  resolve  these  issues  have  been  initiated. 


The  team  has  recorded  data  on  the  implementation  of  practices  in  the 
organization,  and  characterized  the  extent  to  which  practices  in  the  model 
are  implemented. 


Con  tin  ued  on  next  page 
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2.4  Verify  Objective  Evidence  (continued) 


Key  Points  The  data  used  to  formulate  appraisal  outputs  must  be  verified  to  ensure  that 

the  results  of  aggregating  individual  detailed  data  items  will  lead  to 
appropriate  appraisal  outputs. 

Facilitation  techniques  to  guide  the  team  through  difficult  decisions  are 
important  during  this  activity  (as  they  are  during  the  rating  activity  as 
well).  Techniques  to  enhance  the  credibility  of  the  preliminary  findings  are 
also  important. 

Metrics  Planned  versus  actual  effort  expended  for  this  process  (as  with  all 

activities)  will  assist  in  monitoring  progress  as  well  as  planning  subsequent 
appraisals. 

Verification  and  The  appraisal  team  leader  must  ensure  active  participation  in  verification 

Validation  activities  as  a  way  of  confirming  that  the  verification  process  is  working  as 

intended.  Reactions  to  the  validation  activity  (activity  2.5.1,  Validate 
Preliminary  Findings)  will  provide  feedback  to  help  validate  that  this 
activity  was  successful. 

Records  Characterizations  of  practice  implementation  and  strength/weakness 

statements  will  be  recorded  for  subsequent  use  by  the  team. 

Tailoring  The  relative  emphasis  of  mini-team-based  verification  and  verification 

carried  out  by  the  team  as  a  whole  can  be  adjusted  to  meet  the  skills  and 
preferences  of  the  team  at  hand. 

Continued  on  next  page 
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2.4  Verify  Objective  Evidence  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


During  the  conduct  of  an  appraisal,  the  team  gathers  and  analyzes  a  great 
deal  of  detailed  information.  Processes  described  earlier  in  this  document 
clarify  how  data  are  gathered  and  examined.  The  process  described  here  is 
focused  on  understanding  the  information  revealed  by  the  data.  The 
processes  described  after  this  one  are  focused  on  making  reliable  and  valid 
rating  judgments  based  on  the  verified  data. 


The  initial  objective  evidence  provided  by  the  organization  is  used  to 
understand  how  practices  are  intended  to  be  implemented.  Members  of  the 
appraisal  team  then  seek  information  to  confirm  that  the  intended  practices 
are  indeed  implemented.  This  first  verification  activity  (2.4.1,  Verify 
Objective  Evidence)  may  reveal  gaps  in  the  actual  implementation  that  are 
not  apparent  in  the  initial  objective  evidence  provided  by  the  organization. 
The  next  verification  activity  (2.4.2,  Characterize  Implementation  of 
Model  Practices)  then  compares  the  implemented  practices  to  the  practices 
in  the  reference  model.  This  activity  may  also  reveal  gaps  in  the 
implementation(s)  that  will  later  bear  on  the  ratings  assigned  by  the  team. 
Standard  characterizations  to  capture  the  extent  of  practice 
implementation,  first  at  the  instantiation  level  and  then  at  the 
organizational  unit  level,  are  recorded  by  the  team  with  descriptions  of 
gaps  in  implementation. 
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2.4.1  Verify  Objective  Evidence 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  appraisal  team  establishes  a  clear  understanding  of  the  practices 
implemented  in  the  organization.  Typically,  the  organization  provides  a  set 
of  objective  evidence  at  the  beginning  of  the  appraisal  process,  and  the 
team  sets  out  to  verify  the  instances  where  those  practices  are 
implemented.  For  practices  reflecting  project  activities,  the  team  must 
observe  that  each  selected  project  in  the  organizational  scope  has  evidence 
of  implementation.  For  practices  reflecting  organization-level  activities, 
the  team  must  understand  the  support  function  implementation  as  well  as 
any  activities  involving  the  projects  that  indicate  the  implementation  of  the 
practice. 


The  appraisal  team  shall 

•  verify  the  appropriateness  of  direct  artifacts  provided  by  each  project 
or  support  group  for  practices  within  the  appraisal  reference  model 
scope  of  the  appraisal 

•  verify  the  appropriateness  of  indirect  artifacts  provided  by  each 
project  or  support  group  for  practices  within  the  appraisal  reference 
model  scope  of  the  appraisal 

•  verify  the  appropriateness  of  affirmations  provided  by  each  project  or 
support  group  for  practices  within  the  appraisal  reference  model  scope 
of  the  appraisal 

•  verify  that  the  implementation  of  each  model  practice  model  within 
the  appraisal  scope  is  supported  by  direct  artifacts  for  each 
organizational  element  within  the  appraisal  scope  and  corroborated  by 
indirect  artifacts  or  affirmations 

•  obtain  oral  affirmations  corresponding  to  each  specific  and  generic 
goal  within  the  model  scope  of  the  appraisal  for  either  (1)  at  least  one 
project  or  support  group  for  every  associated  practice  and  at  least  one 
practice  for  every  associated  project  or  support  group  for  the  goal 
(i.e.,  1-row,  1 -column),  or  (2)  at  least  50%  of  the  cells  corresponding 
to  the  project-support  group/practice  matrix  for  the  goal 

•  generate  and  verify  preliminary  findings  (i.e.,  statements  describing 
strengths  and/or  gaps  in  the  organizational  unit’s  implemented  practices 
relative  to  practices  defined  in  the  appraisal  reference  model) 


For  practices  implemented  at  the  project  level,  direct  artifacts  and  indirect 
artifacts  or  affirmations  of  practice  implementation  must  be  examined  for 
every  project  sampled  to  represent  the  organizational  unit  being  appraised. 

For  practices  implemented  at  the  organization  level,  direct  artifacts  and 
indirect  artifacts  or  affirmations  of  practice  implementation  are  examined 
in  reference  to  the  organizational  unit  within  the  scope  of  the  appraisal  and 
not  necessarily  for  each  project  sampled.  Aspects  of  the  practice  that  are 
implemented  at  the  project  level  must  be  investigated  for  every  project 
sampled  to  represent  the  organizational  unit. 


Con  tin  ued  on  next  page 
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2.4.1  Verify  Objective  Evidence  (continued) 


Parameters  One  or  more  direct  artifacts  will  be  needed  to  verify  implementation  of 

and  Limits  each  model  practice.  Direct  artifacts  must  be  corroborated  by  either 

(continued)  indirect  artifacts  or  affirmations.  A  description  of  these  indicator  types  is 

contained  in  activity  1.4.1,  Obtain  Initial  Objective  Evidence. 

Oral  affirmations  must  be  provided  via  an  interactive  forum  in  which  the 
appraisal  team  has  control  over  the  interaction. 

Findings  must  be  verified,  that  is,  they  must  be  based  on  corroborated 
objective  evidence  and  they  must  be  consistent  with  other  verified 
findings.  Verified  findings  cannot  be  both  true  and  mutually  inconsistent; 
in  aggregate,  they  constitute  a  set  of  truths  about  the  organizational  unit 
that  must  be  consistent 


Optional  At  the  discretion  of  the  appraisal  team  leader,  verification  of  practices  at 

Practices  the  instantiation  level  may  be  carried  out  solely  by  the  mini-teams.  Team¬ 

wide  review  and  consensus  on  practice  implementation  can  then  focus  on 
the  aggregate-level  characterizations. 

At  the  discretion  of  the  appraisal  team  leader,  the  verification  of  practice 
implementation  at  the  instantiation  level  can  be  reviewed  for  consensus  by 
the  entire  team.  Each  mini-team  provides  an  overview  of  PIIs  for  each 
project  or  support  group  sampled  to  represent  the  organizational  unit. 

A  mix  of  the  two  strategies  above  can  be  used,  selectively  reviewing 
targeted  PAs  in  different  ways,  or  gradually  changing  from  one  strategy  to 
the  other  as  the  team  gains  familiarity  with  the  data  and  the  process. 


Implementation  See  Appendix  C,  Alternative  Practice  Identification  and  Characterization 
Guidance  Guidance,  for  information  on  cases  involving  alternative  practices. 

The  typical  work  products  listed  in  the  reference  models  provide  examples 
of  artifacts  that  can  be  used  as  indicators  of  practice  implementation. 
However,  the  CMMI  appraisal  reference  models  do  not  distinguish 
between  direct  and  indirect  artifacts,  and  these  typical  work  products  are 
examples  only  and  are  not  required;  alternatives  can  be  used  for  both  direct 
and  indirect  artifacts. 

Typically,  much  of  the  objective  evidence  required  to  perform  this 
verification  is  provided  in  advance  of  the  Conduct  Appraisal  phase.  The 
primary  focus  of  data  collection  is  to  permit  the  team  to  verify  that  the 
intended  practices  are  implemented  across  the  organizational  unit.  Where 
the  implemented  practices  differ  from  the  intended  practices,  the  objective 
evidence  provided  at  the  start  of  the  appraisal  process  is  annotated  to  more 
accurately  reflect  the  implemented  process  in  the  organization.  These 
annotations  are  typically  statements  describing  a  gap  in  the  implementation 
of  a  practice,  some  of  which  will  eventually  become  findings. 


Con  tin  ued  on  next  page 
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2.4.1  Verify  Objective  Evidence  (continued) 


Implementation 

Guidance 

(continued) 


Where  gaps  exist  in  the  objective  evidence  provided  in  advance,  the 
appraisal  team  is  forced  to  undertake  data  collection  activities  to  populate 
the  data  set  from  scratch.  An  organization  that  has  a  substantial  process 
improvement  infrastructure  in  place  is  expected  to  have  documented  its 
implementation  of  the  appraisal  reference  model  in  detail.  For 
organizations  with  relatively  little  experience  using  CMMI,  the  cost  of  this 
discovery  process  may  be  so  great  that  undertaking  an  ARC  Class  A 
appraisal,  such  as  SCAMPI  A,  is  not  cost-effective.  For  such  organizations, 
a  Class  B  appraisal  may  be  more  appropriate. 

On  occasion,  the  evidence  obtained  from  the  focus  projects  and  non-focus 
projects  or  the  support  functions  identified  in  the  planned  organizational 
scope  for  the  appraisal  does  not  provide  the  appraisal  team  with  sufficient 
information  to  make  characterization  decisions  about  a  specific  or  generic 
practice.  (See  activity  2.4.2,  Characterize  Implementation  of  Model 
Practices.)  In  these  cases,  evidence  can  be  obtained  from  additional 
projects  or  support  groups.  However,  the  additional  projects  or  support 
groups  need  not  be  included  in  the  oral  affirmation  coverage  requirements 
(see  Required  Practices  above)  for  the  specific  or  generic  goal  associated 
with  the  practice. 

Only  after  team  members  have  a  clear  understanding  of  the  implemented 
practices  can  they  compare  them  to  the  model  to  characterize  the  extent  to 
which  the  organizational  unit  implements  the  practices  in  the  model  or 
acceptable  alternatives.  Artifacts  that  result  from  the  performance  of  the 
practice  should  be  available  for  viewing  by  the  team.  These  artifacts,  as 
well  as  interactions  with  members  of  the  organizational  unit  enacting  the 
practice,  help  to  verify  that  the  practice  was  enacted  as  the  maintainers  of 
the  organizational  process  intended. 


♦ 


CMU/SEI-2006-HB-002 


Page  11-95 


2.4.2  Characterize  Implementation  of  Model  Practices 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 
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Once  a  critical  mass  of  evidence  on  practice  implementation  has  been 
verified,  the  team  (or  mini-team)  turns  to  characterizing  the 
implementation  of  model  practices.  For  each  practice  in  the  model 
included  in  the  selected  scope,  and  each  instance  of  expected  use,  the  team 
will  document  a  characterization  of  the  extent  to  which  the  model  practice 
(or  an  acceptable  alternative)  has  been  implemented.  These  instantiation- 
level  characterizations  are  then  aggregated  to  the  organizational  unit  level. 

Characterizations  of  practice  implementation  are  used  as  a  means  to  focus 
appraisal  team  effort  on  areas  where  professional  judgment  is  needed,  and 
to  aid  in  reaching  team  consensus  on  the  extent  to  which  practices  are 
implemented. 


The  appraisal  team  shall 

•  characterize,  for  each  instantiation,  the  extent  to  which  appraisal 
reference  model  practices  are  implemented 

•  aggregate  practice  implementation  characterization  values  from  the 
instantiation  level  to  the  organizational  unit  level 


The  table  below  summarizes  rules  for  characterizing  instantiation-level 
implementations  of  practices.  Consensus  of  at  least  a  subset  of  appraisal 
team  members  (e.g.,  mini-team  members)  is  necessary  for  instantiation- 
level  characterizations. 

These  rules  apply  to  all  types  of  projects  and  support  groups — focus 
projects,  non-focus  projects,  support  functions,  and  to  any  additional 
sources  from  which  evidence  is  obtained.  Evidence  obtained  from 
additional  sources  must  be  verified  according  to  activity  2.4.1  (Verify 
Objective  Evidence). 


Con  tin  ued  on  next  page 
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2.4.2  Characterize  Implementation  of  Model  Practices  (continued) 


Parameters 
and  Limits 
(continued) 


Label 

Meaning 

Fully  Implemented  (FI) 

•  One  or  more  direct  artifacts  are  present  and  judged  to 
be  adequate, 

•  at  least  one  indirect  artifact  and/or  affirmation  exists  to 
confirm  the  implementation,  and 

•  no  weaknesses  are  noted. 

Largely  Implemented 
(LI) 

•  One  or  more  direct  artifacts  are  present  and  judged  to 
be  adequate, 

•  at  least  one  indirect  artifact  and/or  affirmation  exists  to 
confirm  the  implementation,  and 

•  one  or  more  weaknesses  are  noted. 

Partially  Implemented 
(PI) 

•  Direct  artifacts  are  absent  or  judged  to  be  inadequate, 

•  one  or  more  indirect  artifacts  or  affirmations  suggest 
that  some  aspects  of  the  practice  are  implemented,  and 

•  one  or  more  weaknesses  are  noted. 

OR 

•  One  or  more  direct  artifacts  are  present  and  judged  to 
be  adequate, 

•  no  other  evidence  (indirect  artifacts,  affirmations) 
support  the  direct  artifact(s),  and 

•  one  or  more  weaknesses  are  noted. 

Not  Implemented  (NI) 

•  Direct  artifacts  are  absent  or  judged  to  be  inadequate, 

•  no  other  evidence  (indirect  artifacts,  affirmation) 
supports  the  practice  implementation,  and 

•  one  or  more  weaknesses  are  noted. 

Not  Yet  (NY) 

•  The  project  or  support  group  has  not  yet  reached  the 
stage  in  the  lifecycle  to  have  implemented  the 
practice. 

Con  tin  ued  on  next  page 
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2.4.2  Characterize  Implementation  of  Model  Practices  (continued) 


Parameters 
and  Limits 
(continued) 


The  table  below  summarizes  rules  for  aggregating  instantiation-level 
characterizations  to  derive  organizational  unit-level  characterizations. 
Consensus  of  all  members  of  the  appraisal  team  is  necessary  for 
organizational  unit-level  characterizations. 

These  rules  apply  to  all  types  of  projects  and  support  groups  within  the 
organizational  unit  for  which  implementations  have  been  characterized  in 
accordance  with  the  table  above,  that  is,  to  focus  projects,  non-focus 
projects,  support  functions,  as  well  as  to  additional  sources  from  which 
evidence  has  been  obtained. 

The  column  labeled  “Instantiations”  is  the  input  condition — the  practice 
implementation  characterizations  for  the  set  of  sampled  practice 
instantiations  from  projects  or  support  groups.  The  column  labeled 
“Outcome”  is  the  resultant  aggregated  practice  implementation 
characterization  at  the  organizational  unit  level. 


Instantiations 

Outcome 

Remarks 

All  FI  or  NY, 
with  at  least  one 

FI 

FI 

All  instantiations  are  characterized  FI  or  NY, 
with  at  least  one  FI. 

All  LI  or  FI  or 

NY,  with  at  least 
one  LI 

LI 

All  instantiations  are  characterized  LI  or  FI  or 

NY,  with  at  least  one  LI. 

At  least  one  LI 
or  FI  and  at  least 
one  PI  or  NI 

LI  or  PI 

There  is  at  least  one  instantiation  that  is 
characterized  as  LI  or  FI  and  at  least  one 
instantiation  that  is  characterized  as  PI  or  NI. 

Team  judgment  is  applied  to  choose  LI  or  PI 
based  on  whether  the  weaknesses,  in  aggregate, 
have  a  significant  negative  impact  on  goal 
achievement. 

All  PI  or  NI  or 

NY,  with  at  least 
one  PI 

PI 

All  instantiations  are  characterized  PI  or  NI  or 

NY,  with  at  least  one  PI. 

All  NI  or  NY, 
with  at  least  one 
NY 

NI 

All  instantiations  are  characterized  NI  or  NY, 
with  at  least  one  NI. 

All  NY 

NY 

All  instantiations  are  characterized  NY.  There 
are  no  projects  or  support  groups  within  the 
organizational  unit  that  have  yet  reached  the 
stage  in  the  lifecycle  to  have  implemented  the 
practice.  (NOTE:  If  literally  all  projects  and 
support  groups  in  an  organizational  unit  have 
not  reached  the  stage  in  the  lifecycle  to  have 
implemented  the  practice,  but  will  in  the  future, 
no  rating  can  be  given  for  the  associated  goal 
and  PA.) 

Continued  on  next  page 
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2.4.2  Characterize  Implementation  of  Model  Practices  (continued) 


Optional 

Practices 


While  the  initial  characterization  of  a  practice  implementation  may  be 
proposed  by  a  mini-team  or  some  subset  of  the  team,  the  following 
selections  are  available: 

•  Instantiation- level  characterization  of  practice  implementation  can  be 
reviewed  by  the  entire  team  for  consensus. 

•  Team- wide  review  and  consensus  on  practice  implementation 
characterization  can  be  reserved  for  the  organizational  unit  level. 

•  A  mix  of  the  two  strategies  above,  tailored  to  match  the  learning 
curve  of  the  team  members  or  to  reflect  the  prioritization  of  particular 
PAs,  can  be  used. 


Con  tin  ued  on  next  page 
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2.4.2  Characterize  Implementation  of  Model  Practices  (continued) 


Implementation 

Guidance 


While  written  affirmations  can  be  used  to  confirm  implementation  of 
practices  during  the  characterization  activity  described  above,  oral 
affirmations  are  still  required  to  the  extent  described  in  activity  2.4.1, 
Verify  Objective  Evidence. 

A  weakness  is  defined  in  the  glossary  as  “the  ineffective,  or  lack  of, 
implementation  of  one  or  more  CMMI  model  practices.”  If  there  is  no 
impact  on  the  goal,  there  is  no  need  to  document  a  weakness.  If  the 
appraisal  team  identifies  a  process  improvement  suggestion  with  a  model 
practice  that  is  not  an  ineffective  (or  lack  of)  implementation  of  a  model 
practice,  it  should  be  recorded  as  a  note  rather  than  a  weakness. 

See  Appendix  C,  Alternative  Practice  Identification  and  Characterization 
Guidance,  for  information  on  applying  these  characterization  rules  in 
situations  where  acceptable  alternative  practices  have  been  identified. 

When  the  team  is  ready  to  perform  the  ratings,  these  characterizations 
serve  to  simplify  the  judgments.  The  team  is  then  able  to  focus  on  the 
aggregation  of  weaknesses  observed  to  determine  the  goal  satisfaction 
ratings  (explained  in  process  2.6,  Generate  Appraisal  Results). 
Instantiations  in  situations  where  the  project  or  support  group  has  not  yet 
reached  the  appropriate  lifecycle  phase  where  the  practice  would  be 
enacted  are  characterized  as  Not  Yet  (NY).  The  appraisal-planning 
activities  are  expected  to  prevent  situations  that  severely  limit  the 
examples  of  actual  implementation  for  any  given  practice. 

The  characterization  of  reference  model  practice  implementation  begins  as 
soon  as  sufficient  data  are  available.  It  is  not  necessary  that  data  for  every 
instantiation  be  available  before  the  implementation  of  any  given  practice 
can  be  characterized  at  the  instantiation  level.  However,  before  the 
implementation  of  a  practice  across  the  organizational  unit  can  be 
characterized,  the  instantiation-level  characterizations  have  been 
completed.  Each  instance  of  practice  enactment  is  characterized  using  the 
instantiation-level  characterization  scheme. 

The  characterization  of  practice  implementation  for  the  organizational  unit 
is  carried  out  using  the  aggregation  rules  summarized  in  the  table  above. 
These  rules  provide  a  basis  for  identifying  the  areas  where  professional 
judgment  is  required,  and  simplify  the  areas  where  the  data  are  unanimous. 
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2.5  Validate  Preliminary  Findings 


Purpose 


Entry  Criteria 


Inputs 


Activities 

Outputs 

Outcome 


Exit  Criteria 


Key  Points 


Tools  and 
Techniques 


Validate  preliminary  findings,  including  gaps  in  practice  implementation 
with  members  of  the  organizational  unit.  Exemplary  implementations  of 
model  practices  may  be  highlighted  as  strengths  to  be  included  in  appraisal 
outputs. 


Gaps  in  the  implementation  of  model  practices  have  been  identified,  and 
the  team  has  characterized  the  extent  to  which  model  practices  (or 
acceptable  alternatives  to  those  practices)  have  been  implemented. 
Preliminary  findings  at  the  level  of  the  organizational  unit  have  been 
crafted  and  verified  for  validation. 


•  appraisal  plan,  including  a  schedule  and  participants  for  data 
validation  activities 

•  data  on  practice  implementation,  and  strength/weakness  statements 

•  verified  objective  evidence 

•  characterizations  of  model  practice  implementation 


2.5. 1  Validate  Preliminary  Findings 


•  validated  appraisal  findings 


The  team’s  confidence  in  the  material  that  will  form  the  basis  for  appraisal 
outputs  is  increased,  and  the  process  of  transferring  ownership  of  these 
results  has  been  started.  Any  critical  deficiencies  in  the  data  on  hand  have 
been  identified  and  actions  to  resolve  these  issues  have  been  initiated. 


Preliminary  findings  have  been  validated  with  members  of  the  organization 
who  provided  appraisal  data. 


This  activity  has  one  purpose — ensuring  the  validity  of  the  appraisal  data 
and  associated  outputs.  Managing  the  interaction  with  people  outside  of  the 
team  is  a  vitally  important  process  to  ensure  that  the  results  will  be 
accurate. 


Techniques  to  enhance  the  credibility  of  the  preliminary  findings  are 
important.  Using  a  flip  chart  or  note  taker  during  the  presentation  of 
preliminary  findings  is  often  effective  for  instilling  confidence  among 
audience  members. 


Continued  on  next  page 
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2.5  Validate  Preliminary  Findings  (continued) 

Metrics 

Planned  versus  actual  effort  expended  for  this  process  (as  with  all 
activities)  will  assist  in  monitoring  progress  as  well  as  planning  subsequent 
appraisals.  Gauging  the  level  of  acceptance  for  preliminary  findings  can  be 
facilitated  by  computing  the  percentage  of  findings  adjusted  based  on 
feedback,  then  comparing  this  value  with  past  experience. 

Verification  and 
Validation 

The  attendees  of  preliminary  findings  presentations  are  likely  to  express 
agreement  and/or  discuss  issues  with  the  data  being  validated.  The 
appraisal  team  leader  must  ensure  active  participation  in  these  activities  as 
a  way  of  confirming  that  the  verification  and  validation  process  is  working 
as  intended.  The  actions  taken  following  the  appraisal  will  provide 
feedback  to  help  validate  that  this  activity  was  successful. 

Records 

Characterizations  of  practice  implementation,  strength/weakness 
statements,  and  changes  made  based  on  feedback  will  be  recorded  for 
subsequent  use  by  the  team. 

Tailoring 

Validating  data  is  required,  but  a  variety  of  choices  for  orchestrating  this 
process  are  available.  The  most  common  approach  is  the  preliminary 
findings  presentation.  The  use  of  an  instrument  or  a  more  targeted  focus- 
group  approach  to  validate  preliminary  findings  is  permitted. 

Interfaces 
with  Other 
Processes 

During  the  conduct  of  an  appraisal,  the  team  gathers  and  analyzes  a  great 
deal  of  detailed  information.  Processes  described  earlier  in  this  document 
clarify  how  data  are  gathered  and  examined.  The  process  described  here  is 
focused  on  ensuring  the  data  reflects  actual  conditions  in  the  organization. 
The  processes  described  after  this  one  are  focused  making  reliable  and 
valid  rating  judgments  based  on  the  validated  data. 

Summary  of 
Activities 

When  team  members  have  achieved  their  planned  coverage  of  data 
collection,  the  preliminary  findings  are  validated  with  the  members  of  the 
organization.  This  final  activity  prior  to  rating  allows  team  members  to 
build  confidence  that  their  investigation  has  been  thorough,  and  the 
members  of  the  organization  are  provided  with  an  opportunity  to  correct 
any  perceived  errors  in  the  appraisal  data. 

♦ 
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2.5.1  Validate  Preliminary  Findings 


Activity 

Description 


Required 

Practices 

Parameters 
and  Limits 


In  preparation  for  validating  the  verified  information,  the  appraisal  team 
generates  preliminary  findings  that  summarize  the  practice  implementation 
gaps.  The  preliminary  findings  are  written  in  reference  to  a  single  model 
practice,  and  are  abstracted  to  the  level  of  the  organizational  unit.  The 
statements  should  not  reference  a  specific  individual,  project,  or  other 
identifiable  organizational  sub-unit. 

This  validation  of  preliminary  findings  is  still  primarily  a  data  collection 
activity,  and  the  intent  is  to  validate  the  appraisal  team’s  understanding  of 
the  processes  implemented  within  the  organizational  unit.  Feedback  from 
participants  may  result  in  modifications  to  the  appraisal  team’s  inventory 
of  objective  evidence.  The  results  of  the  validation  activity  are  considered 
in  the  formulation  of  final  findings  and  goal  ratings.  These  latter  activities 
cannot  commence  until  after  the  validation  activity  has  occurred. 


The  appraisal  team  shall 

•  validate  preliminary  findings  with  members  of  the  organizational  unit 


Every  model  practice  characterized  as  Not  Implemented,  Partially 
Implemented,  or  Largely  Implemented  at  the  organizational  unit  level, 
must  have  at  least  one  preliminary  finding  associated  with  it. 

At  least  one  appraisal  participant  from  each  project  or  support  group 
providing  objective  evidence  and  from  any  associated  staff  function  must 
participate  in  the  set  of  validation  activities. 

Only  appraisal  participants  may  participate  (i.e.,  only  people  who  provided 
data  may  participate  in  validation). 

The  minimum  number  of  validation  sessions  required  is  one,  and  no  more 
than  five  are  recommended,  although  no  maximum  limit  is  specified. 

The  rules  of  confidentiality  and  the  expected  use  of  appraisal  data  must  be 
communicated  to  participants  in  each  validation  activity. 
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2.5.1  Validate  Preliminary  Findings  (continued) 


Optional  Preliminary  findings  focused  on  specific  projects,  divisions,  or  other 

Practices  organizational  sub-units  may  be  generated  if  they  are  reflected  in  the 

appraisal  objectives  and  constraints.  This  tailoring  option  also  requires  that 
the  members  of  the  organization  participating  in  the  appraisal  be  fully 
informed  of  the  intended  use  of  the  information  they  provide  to  the 
appraisal  team.  Preliminary  findings  may  include  statements  of  strengths 
for  exemplary  implementations  of  model  practices. 


Implementation  Areas  where  the  appraisal  team’s  inventory  of  objective  evidence  is 

Guidance  insufficient  to  satisfy  the  corroboration  criteria  may  instead  be  addressed 

by  requests  for  additional  information  needed. 

Preliminary  findings  are  the  building  blocks  that  lead  to  the  judgment  of 
goal  satisfaction,  and  are  the  detailed  information  that  forms  the  basis  for 
the  final  findings.  As  an  intermediate  artifact  of  the  appraisal  process, 
preliminary  findings  are  used  to  ensure  traceability  between  appraisal 
inputs  and  appraisal  outputs. 

Feedback  from  participants  on  the  preliminary  findings  should  be  solicited 
by  the  appraisal  team  and  considered  for  possible  revisions  to  its  inventory 
of  objective  evidence. 

If  virtual  methods  such  as  video  conferences,  teleconferences,  and  other 
similar  technology  are  used  for  the  validation  session(s),  the  appraisal  team 
leader  should  ensure  that  these  methods  in  no  way  compromise  the 
integrity  of  the  appraisal  or  the  accuracy  of  the  results.  Virtual  methods 
should  allow  for  adequate  interaction  between  the  appraisal  team  members 
and  the  appraisal  participants  and  should  provide  mechanisms  for  the 
appraisal  team  to  control  the  validation  session.  Appropriate  arrangements, 
such  as  meeting  rooms  at  remote  sites,  should  be  made  and  checks  done  to 
ensure  only  appraisal  participants  attend  this  presentation  to  maintain 
confidentiality. 

It  is  not  expected  that  preliminary  findings  will  provide  a  detailed  listing  of 
the  implementation  status  of  every  model  practice  in  every  sampled  project 
or  support  group.  Furthermore,  it  is  not  expected  that  the  preliminary 
findings  will  identify  the  status  of  individual  projects  or  support  groups 
with  regard  to  practice  implementation  or  goal  achievement.  An  appraisal 
sponsor  may  request  these  more  detailed  appraisal  results.  The  appraisal 
team  leader  should  negotiate  for  the  proper  allocation  of  time  to 
accommodate  this  tailoring  option,  and  the  expectation  that  such 
information  will  be  preserved  at  the  end  of  the  appraisal  should  be  made 
clear  to  all  appraisal  participants. 
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2.5.1  Validate  Preliminary  Findings  (continued) 


Implementation 

Guidance 

Preliminary 

Findings 

Presentations 


Implementation 

Guidance 

Focus  Croups 


Implementation 

Guidance 

Survey 

Instrument 


An  interactive  presentation  is  the  most  effective  mechanism  for  validating 
the  preliminary  findings.  The  members  of  the  organization  who  provided 
data  to  the  appraisal  team  are  typically  brought  together  in  a  conference 
room,  and  a  slide  presentation  is  used  to  review  the  preliminary  findings  in 
an  effort  to  invite  people  to  provide  additional  data  or  express  their 
agreement  with  the  summary  statements.  The  audience  is  often  grouped  by 
seniority  in  the  organization,  and  separate  presentations  are  made  for 
practitioners,  project  managers,  and  middle  managers. 

During  the  presentation,  one  or  more  members  of  the  team  review  the 
preliminary  findings  statements  and  provide  the  audience  with  an 
opportunity  to  comment  or  ask  questions.  The  presenter  uses  only  the 
words  crafted  by  the  appraisal  team  and  avoids  elaborating  on  the  findings 
using  his/her  own  words.  When  questions  are  asked  about  a  preliminary 
finding,  the  appraisal  team  leader  provides  any  clarification  needed  to 
understand  what  the  statement  means.  However,  team  members  avoid  the 
appearance  that  they  are  justifying  the  content  of  the  statement. 

The  detailed  data  that  led  to  the  preliminary  findings  must  be  protected, 
and  negotiations  for  wording  changes  avoided.  The  appraisal  team  records 
new  data  made  available  to  them  without  commenting  on  how  the  data 
may  be  interpreted  or  how  the  findings  may  need  to  change. 


As  an  alternative  (or  in  addition)  to  the  presentation,  focus  groups  can  be 
used  to  probe  more  deeply  into  specific  areas  of  the  reference  model  with  a 
targeted  audience.  This  use  of  focus  groups  permits  the  team  to  explore  a 
particular  area  in  more  depth  to  help  shaipen  the  appraisal  results,  or  to 
raise  the  visibility  of  the  results  to  people  who  are  most  informed  on  the 
topic.  For  example,  a  focus  group  conducted  with  project  managers  could 
be  an  ideal  environment  to  validate  (and  gather  more  detailed  data  on)  the 
topic  of  project  planning  and  project  monitoring.  In  contrast,  a  focus  group 
composed  of  Engineering  Process  Group  (EPG)  members  may  be  an  ideal 
setting  to  validate  findings  associated  with  the  organization’s  infrastructure 
for  process  improvement.  The  preliminary  findings  that  relate  to  the  group 
may  be  distributed  as  handouts  or  displayed  using  a  projector,  and  the 
participants  can  engage  in  a  free-form  dialogue  with  the  team  and  amongst 
themselves.  Notes  taken  by  the  members  of  the  team  are  treated  as  any 
data  collected  during  an  interview  would  be. 


Finally,  a  survey  instrument  can  be  used  in  addition  (or  as  an  alternative) 
to  either  of  the  techniques  above.  A  carefully  worded  instrument  that  asks 
respondents  to  rate  their  level  of  agreement  with  the  finding  statement,  and 
provides  an  opportunity  for  written  feedback,  can  provide  a  low-cost  and 
timely  source  of  data  for  the  team. 
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2.6  Generate  Appraisal  Results 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Rate  goal  satisfaction  based  on  the  extent  of  practice  implementation 
throughout  the  organizational  scope  of  the  appraisal.  The  extent  of  practice 
implementation  is  judged  based  on  validated  data  (e.g.,  direct,  indirect,  and 
affirmation  objective  evidence)  collected  from  the  entire  representative 
sample  of  the  organizational  unit.  The  rating  of  capability  levels  and/or 
maturity  levels  is  driven  by  the  goal  satisfaction  ratings. 


The  set  of  validated  preliminary  findings,  statements  of  practice 
implementation  gaps,  and/or  tabulations  of  validated  objective  evidence  of 
practice  implementation  on  which  they  are  based  are  available.  Team 
members  are  confident  that  they  have  obtained  all  the  pertinent  data  they 
must  make  rating  judgments.  The  data  obtained  completely  covers  the 
practices  within  the  defined  appraisal  reference  model  scope  and  the  entire 
representative  sample  selected  for  the  organizational  unit. 


appraisal  data 

•  validated  preliminary  findings 

•  tabulations  of  objective  evidence  of  practice  implementation 

•  annotated  worksheets,  checklists,  working  notes 


2.6.1  Derive  Findings  and  Rate  Goals 
2.6.2a  Determine  Process  Area  Capability  Level 
2.6.2b  Determine  Satisfaction  of  Process  Areas 
2.6.3a  Determine  Capability  Profile 
2.6.3b  Determine  Maturity  Level 
2.6.4  Document  Appraisal  Results 


•  final  findings 

•  recorded  rating  decisions 


A  formal  rating  decision  for  each  appraisal  reference  model  component 
that  was  planned  to  be  rated,  and  for  which  the  team  obtained  complete  or 
sufficient  data 


Ratings  against  all  components  per  the  plan  have  been  made  and  recorded. 
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2.6  Generate  Appraisal  Results  (continued) 

Key  Points  The  judgment  of  goal  satisfaction  is  based  on  and  traceable  to  the  extent  of 

the  implementation  of  practices  associated  with  that  goal  (or  alternative 
practices  contributing  equivalently  to  goal  satisfaction). 

Success  in  this  activity  is  driven  by  team  members’  ability  to  limit  their 
focus  to  the  data  that  support  the  judgments,  and  to  avoid  issues  that 
threaten  their  ability  to  be  objective.  This  activity  can  create  a  great  deal  of 
stress  for  team  members  under  pressure  to  help  their  organization  “do 
well.”  The  appraisal  team  leader  must  skillfully  facilitate  this  activity  when 
external  pressures  exist. 

Tools  and  There  is  a  significant  amount  of  data  to  review  in  making  each  round  of 

Techniques  judgments.  Rating  worksheets  and  automated  support  tools  facilitate  the 
team’s  decision-making  process  by  presenting  necessary  data  in  a  concise, 
well-organized  manner.  When  controversial  issues  are  encountered,  the 
appraisal  team  leader  must  actively  facilitate  to  ensure  that  the  team 
remains  focused  on  the  pertinent  issues.  Strategic  rest  breaks,  and 
sequencing  and  pacing  critical  discussions,  are  often  keys  to  success. 

Metrics  •  planned  versus  actual  effort  for  each  component  rated 

•  number  of  model  components  rated  satisfied  or  unsatisfied 

Verification  and  The  appraisal  team  leader  verifies  that  the  rating  process  was  performed  in 

Validation  accordance  with  the  method  rules  and  the  rating  baseline  selected  and 

documented  in  the  appraisal  plan.  Work  aids  used  to  record  the  team 
judgments  help  ensure  traceability  to  the  basis  for  the  rating  judgments. 

Records  A  worksheet  or  other  work  aid  may  be  used  to  make  a  record  of  the  rating 

decisions.  A  process  area  profile  is  often  an  effective  means  of  recording 
and  communicating  these  results. 

Tailoring  The  method  provides  tailoring  options  for  rating  additional  model 

components.  The  minimum  requirement  is  to  rate  the  specific  and  generic 
goals  associated  with  each  PA  in  the  scope  of  the  appraisal.  In  addition,  the 
sponsor  may  request  that  maturity  level  and/or  capability  level  ratings  be 
performed  and  reported.  Through  negotiation  between  the  appraisal  team 
leader  and  the  appraisal  sponsor,  a  decision  to  rate  individual  practices  can 
also  be  made. 
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2.6  Generate  Appraisal  Results  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


The  rating  judgments  made  by  the  appraisal  team  members  are  dependent 
on  the  quality  of  the  data  available  to  them,  as  well  as  their  ability  to 
reliably  judge  the  implementation  and  institutionalization  of  practices  in 
the  organization  that  relate  to  the  reference  model.  All  the  processes 
previously  described  contribute  to  the  team’s  ability  to  effectively  execute 
this  process.  The  Analyze  Requirements  process  establishes  the  rating 
baseline,  the  organizational  unit  to  which  ratings  will  apply,  and  the 
purpose  for  which  the  ratings  will  be  used.  The  Develop  Appraisal  Plan 
process,  in  conjunction  with  the  Obtain  and  Inventory  Initial  Objective 
Evidence  and  Prepare  for  Appraisal  Conduct  processes,  determines  the 
sample  of  the  organizational  unit  for  which  data  will  be  collected  and  from 
which  the  ratings  will  be  determined.  The  Select  and  Prepare  Team  process 
ensures  that  the  team  has  sufficient  knowledge  and  skills  to  interpret  the 
data  and  arrive  at  sound  rating  judgments.  The  Examine  Objective 
Evidence  and  Document  Objective  Evidence  processes  provide  the  basic 
information  that  is  needed  to  support  judgments  in  a  form  that  facilitates 
making  the  judgments.  The  Verify  Objective  Evidence  process 
characterizes  the  extent  to  which  the  organizational  unit  implements 
practices  in  the  model  (or  acceptable  alternatives).  The  Validate 
Preliminary  Findings  process  validates  findings  describing  any  weaknesses 
associated  with  the  practice  implementations.  Upon  the  successful 
execution  of  these  processes,  the  team  is  ready  to  rate  the  satisfaction  of 
goals  dependent  on  those  practices. 


The  required  and  fundamental  rating  activity  involves  making  team 
judgments  about  goal  satisfaction  for  each  and  every  specific  and  generic 
goal  within  the  appraisal  reference  model  scope  defined  in  the  rating 
baseline.  Once  goal  satisfaction  has  been  determined,  optional  rating 
activities  can  be  performed  in  accordance  with  the  defined  rating  baseline 
and  the  selected  model  representation(s)  (i.e.,  continuous,  staged,  or  both). 
The  first  optional  activity  focuses  on  rolling  up  goal  satisfaction  to  PA 
ratings.  The  team  determines  a  PA  capability  level  rating  (0  through  5)  for 
each  PA  in  the  continuous  representation  that  is  within  the  appraisal  scope, 
and/or  the  team  determines  a  Satisfied/Unsatisfied  rating  for  each  PA  in 
the  staged  representation  that  is  within  the  appraisal  scope.  The  second 
optional  activity  continues  the  rating  roll  up  to  cover  all  PAs  within  the 
selected  reference  model  scope.  In  the  case  of  the  continuous 
representation  the  team  creates  a  profile  showing  the  capability  levels  for 
all  PAs  considered.  The  profile  can  then  be  used  to  compute  a  maturity 
level  through  the  equivalent  staging  described  in  the  model.  In  the  case  of 
the  staged  representation,  the  team  assigns  a  maturity  level  rating  (1 
through  5)  corresponding  to  the  highest  level  in  the  model  for  which  all 
applicable  PAs  have  been  rated  as  satisfied.  The  optional  activities 
described  in  2.6.2a  and  2.6.3a  cover  the  continuous  representation;  those  in 
2.6.2b  and  2.6.3b  cover  the  staged  representation.  As  indicated,  these 
options  are  not  mutually  exclusive. 
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2.6.1  Derive  Findings  and  Rate  Goals 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  judgments  made  about  goal  satisfaction  are  driven  by  the  findings  that 
were  documented  by  the  appraisal  team  and  validated  by  appraisal 
participants  as  well  as  the  extent  of  implementation  of  associated  practices. 
The  preliminary  findings  focus  on  gaps  in  the  implementation  of  practices. 
When  performing  goal  ratings,  the  team  judges  whether  or  not  these  gaps 
in  the  implementation  of  practices  (in  aggregate)  threaten  the 
organizational  unit’s  ability  to  satisfy  the  goals  associated  with  the 
practices. 


The  appraisal  team  shall 

•  derive  final  findings  using  preliminary  findings  statements,  feedback 
from  validation  activities,  and  any  additional  objective  evidence 
collected  as  a  result  of  the  validation  activities 

•  rate  each  specific  goal  and  generic  goal  within  the  reference  model 
scope  of  the  appraisal,  based  on  the  practice  implementation 
characterizations  at  the  organizational  unit  level  as  well  as  the 
aggregation  of  weaknesses  associated  with  that  goal 

•  obtain  appraisal  team  consensus  on  the  findings  statements  and 
ratings  generated  for  the  organizational  unit  level 


When  deriving  final  findings,  the  aim  is  to  create  goal-level  statements  that 
summarize  the  gaps  in  practice  implementation.  These  statements  must  be 
abstracted  to  the  level  of  the  organizational  unit,  and  cannot  focus  on 
individual  projects  or  increments  (unless  the  tailoring  option  for  project-  or 
increment-specific  findings  has  been  agreed  on  during  planning). 

A  goal  must  be  rated  Not  Rated  if  there  are  any  associated  practices  that 
are  not  characterized  at  the  organizational  unit  level  or  that  are 
characterized  as  Not  Yet  at  the  organizational  unit  level. 

A  goal  is  rated  Not  Rated  if  the  associated  set  of  objective  evidence  does 
not  meet  the  defined  criteria  for  sufficient  data  coverage. 

The  goal  is  rated  Satisfied  if  and  only  if 

•  all  associated  practices  are  characterized  at  the  organizational  unit 
level  as  either  Largely  Implemented  or  Fully  Implemented,  and 

•  the  aggregation  of  weaknesses  associated  with  the  goal  does  not  have 
a  significant  negative  impact  on  goal  achievement. 

For  a  goal  to  be  rated  as  Unsatisfied,  the  team  must  be  able  to  describe 
how  the  set  of  documented  weaknesses  (or  single  weakness)  led  to  this 
rating. 


Con  tin  ued  on  next  page 
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2.6.1  Derive  Findings  and  Rate  Goals  (continued) 


Optional  Final  findings  may  include  statements  of  strengths  for  exemplary 

Practices  implementations  of  model  practices. 


Implementation  Any  endeavor  that  results  in  producing  a  score,  grade,  or  rating  is  by 
Guidance  definition  an  area  of  sensitivity  to  those  affected  by  the  outcome.  An 

objective  and  clear-cut  basis  for  assigning  a  rating  lessens  this  sensitivity 
and  results  in  a  more  consistent  basis  of  comparison  among  the 
organizational  units  and  goals  rated.  Judgments  made  prior  to  and  during 
the  rating  process  should  be  based  on  observable  facts  and  should  be  made 
at  the  lowest  level  of  abstraction  that  makes  sense.  In  the  case  of  SCAMPI 
A  appraisals,  the  lowest  level  of  abstraction  is  characterizing  the  extent  of 
practice  implementation  for  each  project  or  support  group  within  the 
representative  sample.  Characterizations  made  at  the  instantiation  level  are 
aggregated  into  a  characterization  of  the  extent  of  practice  implementation 
throughout  the  organizational  unit,  as  described  earlier  in  process  2.4, 
Verify  Objective  Evidence.  The  judgment  of  goal  satisfaction  is  then  based 
on,  and  directly  traceable  to,  the  extent  of  implementation  of  practices 
associated  with  that  goal.  (See  Appendix  C,  Alternative  Practice 
Identification  and  Characterization  Guidance,  for  information  regarding 
the  judgment  of  goal  satisfaction  when  acceptable  alternative  practices  are 
involved.) 

The  use  of  informative  material  in  the  appraisal  reference  model  to  form  a 
checklist  is  explicitly  discouraged. 

Findings  should  be  phrased  in  terms  that  best  facilitate  decision  making  by 
the  appraisal  sponsor  and  taking  action  on  the  appraisal  results. 


♦ 
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2.6.2a  Determine  Process  Area  Capability  Level 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 

Implementation 

Guidance 


When  using  the  continuous  representation  of  the  appraisal  reference 
model,  the  team  may  make  rating  judgments  about  each  PA  (and 
associated  capability  level)  within  the  scope  of  the  appraisal.  Assigning 
capability  level  ratings  is  an  optional  activity,  selected  at  the  discretion  of 
the  appraisal  sponsor  and  documented  in  the  appraisal  input. 


The  appraisal  team  shall 

•  rate  the  capability  levels  for  each  PA  within  the  scope  of  the  appraisal, 
based  on  the  highest  level  and  all  levels  below  for  which  its  specific 
goals  and  the  generic  goals  within  the  appraisal  scope  have  been 
satisfied  (if  this  rating  option  was  selected  during  planning). 

•  if  any  goals  for  a  PA  are  rated  Not  Rated,  the  PA  will  be  rated  Not  Rated 
and  a  capability  level  will  not  be  assigned. 

•  when  a  PA  is  determined  to  be  outside  of  the  organizational  unit’s  scope 
of  work,  the  PA  is  designated  as  “not  applicable”  and  is  not  rated,  and  a 
capability  level  will  not  be  assigned. 

•  when  an  applicable  PA  is  outside  of  the  scope  of  the  model  used  for  the 
appraisal,  the  PA  is  designated  as  “out  of  scope”  and  is  not  rated,  and  a 
capability  level  will  not  be  assigned. 


Capability 

Level 

Process  Areas 

0 

Default  Rating 

1 

Generic  goal  for  capability  level  1  is  rated  Satisfied. 

All  specific  goals  are  rated  Satisfied. 

2 

Generic  goals  for  capability  levels  1  and  2  are  rated  Satisfied. 

All  specific  goals  are  rated  Satisfied. 

3 

Generic  goals  for  capability  levels  1,  2,  and  3  are  rated  Satisfied. 

All  specific  goals  are  rated  Satisfied. 

4 

Generic  goals  for  capability  levels  1,  2,  3,  and  4  are  rated  Satisfied. 

All  specific  goals  are  rated  Satisfied. 

5 

Generic  goals  for  capability  levels  1,  2,  3,  4,  and  5  are  rated 

Satisfied. 

All  specific  goals  are  rated  Satisfied. 

None 


None 
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2.6.2b  Determine  Satisfaction  of  Process  Areas 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


Implementation 

Guidance 


When  using  the  staged  representation  of  the  appraisal  reference  model,  the 
team  may  derive  the  satisfaction  of  PAs  from  the  set  of  goal  satisfaction 
judgments.  Assigning  PA  satisfaction  ratings  is  an  optional  activity, 
selected  at  the  discretion  of  the  appraisal  sponsor  and  documented  in  the 
appraisal  input. 


The  appraisal  team  shall 

•  rate  the  satisfaction  of  each  PA  in  the  scope  of  the  appraisal  based  on 
the  ratings  of  the  goals  within  each  PA,  if  this  rating  option  was 
selected  during  planning. 


PAs  must  be  assigned  rating  values  of  Satisfied,  Unsatisfied,  Not 
Applicable,  Out  of  Scope,  or  Not  Rated. 

A  PA  is  rated  Satisfied  if  and  only  if  all  of  its  specific  and  generic  goals 
associated  with  a  given  maturity  level  and  below  are  rated  Satisfied. 

If  even  one  of  the  goals  associated  with  a  given  maturity  level  in  a  PA  is 
rated  Unsatisfied,  then  the  PA  is  rated  Unsatisfied  for  that  maturity  level  and 
above. 

When  a  PA  is  determined  to  be  outside  of  the  organizational  unit’s  scope 
of  work,  the  PA  is  designated  as  Not  Applicable  and  is  not  rated.  The 
identification  of  a  PA  as  Not  Applicable  must  occur  during  the  planning  of 
the  appraisal. 

When  an  applicable  PA  is  outside  the  scope  of  the  appraisal  reference 
model  used  for  the  appraisal,  the  PA  is  designated  as  Out  of  Scope  and  is 
not  rated. 

If  even  one  of  the  goals  in  a  PA  is  rated  Not  Rated  and  none  of  the  other 
goals  are  rated  Unsatisfied,  then  the  PA  is  rated  Not  Rated. 


A  profile  to  summarize  the  satisfaction  of  goals  may  be  created  to  provide 
further  insight  about  the  rating  outcomes.  Where  a  PA  is  rated  as 
Unsatisfied,  this  more  detailed  view  of  the  rating  outcomes  may  provide 
focus  and  visibility  at  a  lower  level  of  detail. 


PA  satisfaction  is  a  direct  function  of  goal  satisfaction.  A  PA  is  rated  as 
Satisfied  if  every  goal  contained  in  the  PA  is  rated  as  Satisfied.  A  PA  is 
rated  as  Unsatisfied  if  any  goal  is  rated  as  Unsatisfied.  This  guideline 
ensures  that  one  or  more  weaknesses  exist  that  serve  to  explain  why  the 
goal,  and  therefore  the  PA,  are  not  satisfied. 

PA  ratings  need  not  be  reported  to  appraisal  participants,  if  the  sponsor 
does  not  wish  to  disclose  these  results.  However,  a  documented  output 
from  this  rating  activity,  if  it  is  performed,  is  a  required  component  in  the 
appraisal  record  as  well  as  the  ADS. 


♦ 
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2.6.3a  Determine  Capability  Profile 


Activity  When  using  the  continuous  representation  of  the  appraisal  reference 

Description  model,  the  team  may  determine  a  Capability  Profde  that  graphically 

depicts  the  capability  level  ratings  assigned  to  each  PA  within  the  scope  of 
the  appraisal.  The  generation  of  a  Capability  Profile  is  an  optional  activity, 
selected  at  the  discretion  of  the  appraisal  sponsor  and  documented  in  the 
appraisal  input. 

The  appraisal  team  shall 

•  generate  a  Capability  Profile  depicting  the  capability  level  attained  for 
each  PA  within  the  scope  of  the  appraisal,  if  this  rating  option  was 
selected  during  planning 

Parameters  A  simple  bar  chart  is  used  for  this  display.  Each  PA  is  represented  in  a 

and  Limits  single  bar  along  the  horizontal  axis,  and  the  vertical  axis  represents  the 

capability  level  dimension.  The  height  of  each  bar  communicates  the 
capability  level  of  the  PA  represented. 

Capability  levels  take  only  the  values  0,  1,  2,  3,  4,  or  5.  Intermediate  values 
(e.g.,  2.7)  are  not  defined  for  this  appraisal  outcome,  and  any 
embellishment  of  the  Capability  Profile  with  such  values  is  outside  the 
boundaries  of  SCAMPI  A. 

A  profile  to  summarize  the  satisfaction  of  goals  may  be  created  to  provide 
further  insight  about  the  rating  outcomes.  In  situations  where  a  PA 
capability  level  rating  does  not  reflect  the  desired  outcome,  this  more 
detailed  view  may  provide  focus  and  visibility  at  a  lower  level  of  detail. 

The  reference  model  provides  for  equivalent  staging,  whereby  a  Capability 
Profile  can  be  used  to  derive  an  equivalent  maturity  level  rating  (see 
activity  2.6.3b,  Determine  Maturity  Level). 

Implementation  A  presentation  template  referred  to  as  a  Capability  Profile  is  typically  used 
Guidance  to  communicate  the  aggregate  level  rating  results  to  the  sponsor  and  others 

designated  by  the  sponsor. 

Comparing  different  PAs  with  respect  to  their  relative  capability  level 
ratings  may  be  informative  in  discussing  trends  or  patterns  in  the 
organization. 

This  activity  may  be  omitted  entirely,  as  it  is  a  tailoring  option.  If  a 
Capability  Profile  is  to  be  derived,  the  ratings  reflected  in  the  profile  are 
derived  as  described  in  activity  2.6.2a,  Determine  Process  Area  Capability 
Level. 

♦ 


Optional 

Practices 


Required 

Practices 
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2.6.3b  Determine  Maturity  Level 

Activity 

Description 

Historically,  one  of  the  most  visible  outcomes  of  an  appraisal  has  been  the 
maturity  level  rating  assigned.  The  determination  of  a  maturity  level  rating 
is  straightforward,  and  is  derived  mechanically  from  the  ratings  assigned  at 
the  lower  levels  of  detail.  Assigning  a  maturity  level  rating  is  an  optional 
activity,  selected  at  the  discretion  of  the  appraisal  sponsor  and  documented 
in  the  appraisal  input. 

Required 

Practices 

The  appraisal  team  shall 

•  rate  the  maturity  level  based  on  the  ratings  assigned  to  PAs,  if  this 
rating  option  was  selected  during  planning 

Parameters 
and  Limits 

When  using  the  staged  representation,  the  maturity  level  determined  is  the 
highest  level  at  which  all  PAs  contained  within  the  maturity  level,  and 
within  all  lower  maturity  levels,  are  rated  as  Satisfied  or  Not  Applicable. 
The  single  exception  to  this  rule  is  that  generic  goal  3  for  applicable 
maturity  level  2  PAs  must  also  be  rated  Satisfied  for  a  maturity  level  rating 
of  3  or  higher  to  be  determined. 

When  using  the  continuous  representation,  the  appraisal  reference  model 
provides  for  equivalent  staging,  whereby  a  Capability  Profile  can  be  used 
to  derive  an  equivalent  maturity  level  rating.  A  maturity  level  for  a 
continuous  representation  is  achieved  if  the  Capability  Profile  is  at  or 
above  the  target  profile  for  all  PAs  for  that  maturity  level  and  all  lower 
maturity  levels  in  the  equivalent  staging,  excepting  those  PAs  that  are 
designated  as  Not  Applicable.  The  equivalence  of  particular  Capability 
Profiles  and  particular  maturity  levels  is  addressed  in  Chapter  3  of  the 
reference  model. 

To  determine  a  maturity  level  as  an  output  of  the  appraisal,  the  model 
scope  of  the  appraisal  must  include  the  minimum  set  of  PAs  required  by 
the  appraisal  reference  model. 

Optional 

Practices 

None 

Implementation 

Guidance 

This  activity  may  be  omitted  entirely,  as  it  is  a  tailoring  option.  If  a 
maturity  level  is  to  be  reported,  the  PA  ratings  that  form  the  basis  for  the 
maturity  level  rating  are  derived  as  described  in  activity  2.6.2b,  Determine 
Satisfaction  of  Process  Areas. 

♦ 
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2.6.4  Document  Appraisal  Results 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 

Implementation 

Guidance 


The  results  of  the  appraisal  conduct  must  be  documented  for  reporting. 
Verbal  reports  of  the  rating  outcomes  or  oral  explanations  of 
implementation  gaps  discovered  by  the  team  are  not  sufficient  to 
communicate  appraisal  results. 


The  appraisal  team  shall 

•  document  the  final  findings 

•  document  the  rating  outcome(s) 

•  document  the  ADS 


The  ADS  and  the  set  of  appraisal  outputs  agreed  on  with  the  appraisal 
sponsor  must  be  documented. 

Regardless  of  the  needs  of  the  sponsor,  the  ADS,  the  goal  ratings,  and  the 
associated  findings  must  be  documented  as  a  part  of  the  appraisal 
information  returned  to  the  CMMI  Steward. 


Any  optional  outputs  requested  by  the  appraisal  sponsor  are  also  created 
during  this  activity. 


This  activity  is  focused  on  collecting  and  documenting  the  results  of  prior 
activities  related  to  the  generation  of  findings  and  ratings.  Depending  on 
the  planned  recipients  of  the  results,  multiple  forms  of  the  results  may  be 
needed.  Certain  data  may  not  be  appropriate  for  all  audiences,  or  the  style 
or  language  of  the  results  may  need  to  be  adjusted  to  best  fit  the  needs  of 
the  recipients. 

The  documented  appraisal  results  are  typically  provided  in  a  final  findings 
presentation,  described  in  activity  3.1.1,  Deliver  Final  Findings. 


♦ 
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3  Report  Results 


3.1  Deliver  Appraisal  Results 

Purpose  Provide  credible  appraisal  results  that  can  be  used  to  guide  actions. 

Represent  the  strengths  and  weaknesses  of  the  processes  in  use  at  the  time. 
Provide  ratings  (if  planned  for)  that  accurately  reflect  the  capability  level 
or  maturity  level  of  the  processes  in  use. 

Entry  Criteria  •  Objective  evidence  has  been  verified  (through  the  team  process). 

•  Preliminary  findings  have  been  validated. 

•  Ratings  have  been  determined  (for  model  components  selected  for 
rating). 

•  Final  findings  have  been  created  and  reviewed  by  the  team. 

Inputs  •  appraisal  data 

-  final  findings 

-  ratings 

•  appraisal  artifacts 

-  appraisal  input 

-  appraisal  plan 

Activities  3.1.1  Deliver  Final  Findings 

3.1.2  Conduct  Executive  Session(s) 

3.1.3  Plan  for  Next  Steps 

Outputs  •  documented  final  findings 

•  final  report  (if  requested) 

•  recommendations  report  (if  requested) 

Outcome  •  The  sponsor  and  the  appraised  organizational  unit  are  provided  with 

the  results  of  the  appraisal. 

•  A  valid  and  reliable  characterization  of  the  current  state  of  the 
processes  in  use  across  the  organizational  unit  is  documented. 

Exit  Criteria  •  Appraisal  results  are  delivered  to  the  appraisal  sponsor  and 

organizational  unit. 

•  An  executive  session  is  conducted,  if  appropriate. 

Continued  on  next  page 
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3.1  Deliver  Appraisal  Results  (continued) 


Key  Points  The  appraisal  results  are  intended  to  support  decision  making,  and  should 

be  delivered  in  a  way  that  promotes  appropriate  actions.  Whether  the 
appraisal  was  conducted  for  internal  process  improvement,  supplier 
selection,  or  process  monitoring  purposes,  the  delivery  of  results  should 
facilitate  the  actions  that  will  be  driven  by  the  information. 

Tools  and  Templates  containing  standard  information  for  use  in  a  final  findings 

Techniques  briefing  are  provided  to  all  SCAMPI  Lead  Appraisers.  Experienced 

appraisal  team  leaders  frequently  use  electronic  (database)  tools  that 
support  the  transformation  of  raw  appraisal  data  into  appraisal  results. 
These  tools  may  be  useful  in  real  time  as  appraisal  results  are  presented. 
Strategies  for  presenting  and  packaging  the  results  should  leverage 
presentation  and  documentation  techniques  that  best  suit  the  audience. 

Metrics  It  is  highly  recommended  that  the  attendance  at  the  final  briefing  (if  one  is 

held)  be  recorded.  Significant  absenteeism  of  key  stakeholders  is  likely  to 
be  an  indication  of  risk  for  future  success  in  addressing  the  appraisal 
findings. 

The  required  elements  of  appraisal  results  are  specified  in  the  activity 
description  found  here,  and  a  checklist  can  support  verification  that  these 
elements  are  present.  Validation  of  this  activity  can  only  occur  after  the 
appraisal  is  complete. 

Records  •  final  findings 

•  final  report  (if  requested) 

•  recommendations  report  (if  requested) 

Tailoring  If  the  method  is  being  used  as  part  of  a  supplier  selection  process,  there 

may  be  acquisition  regulations  or  limitations  that  constrain  the  mechanisms 
used  to  deliver  appraisal  results  to  the  appraised  organization. 

In  some  internal  process  improvement  usages  of  the  method,  the  executive 
session  may  be  tailored  out.  The  appraisal  sponsor  should  make  this 
decision,  with  the  full  involvement  of  the  appraisal  team  leader. 

Continued  on  next  page 
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3.1  Deliver  Appraisal  Results  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


Upon  completion  of  the  Generate  Appraisal  Results  process,  the  ratings 
and  findings  generated  are  used  to  prepare  and  deliver  the  final  appraisal 
results  to  the  appraisal  sponsor  and  organizational  unit.  The  appraisal 
results  become  part  of  the  appraisal  record,  which  is  discussed  in  process 
3.2,  Package  and  Archive  Appraisal  Assets. 


The  final  findings  contain  the  validated  strengths,  weaknesses,  and  ratings 
(as  defined  by  the  appraisal  plan),  reflecting  the  organizational  process 
capability  and/or  maturity  level  for  PAs  within  the  appraisal  scope.  Other 
appraisal  outputs,  as  requested  by  the  appraisal  sponsor  and  documented  in 
the  appraisal  plan,  are  generated  and  provided.  Optionally,  a  separate 
executive  session  may  also  be  held  to  clarify  and  discuss  the  appraisal 
results  from  a  senior  management  perspective  that  facilitates  decision 
making.  Plans  are  established  for  acting  on  the  appraisal  results. 


♦ 


Page  11-118 


CMU/SEI-2006-HB-002 


3.1.1  Deliver  Final  Findings 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


The  final  findings  contain  a  summary  of  the  strengths  and  weaknesses  for 
each  PA  within  the  appraisal  scope,  as  well  as  additional  information  that 
provides  context  for  the  findings.  The  generation  of  the  findings  is 
addressed  in  activity  2.6.1,  Derive  Findings  and  Rate  Goals;  this  activity 
relates  to  the  delivery  of  these  findings  to  the  appraisal  sponsor  and 
appraised  organization.  These  findings  may  be  in  a  summarized  form,  with 
the  detailed  findings  provided  as  backup  information,  and  is  often 
presented  using  view  graphs  in  a  meeting  room  or  auditorium. 

In  addition  to  the  final  findings,  a  draft  ADS  summarizing  the  results  of  the 
appraisal  is  provided  to  the  appraisal  sponsor. 


The  appraisal  team  shall 

•  provide  appraisal  final  findings,  signed  by  the  appraisal  team  leader 
and  all  appraisal  team  members,  to  the  appraisal  sponsor 

•  provide  an  ADS  to  the  appraisal  sponsor  summarizing  the  appraisal 
results  and  conditions  under  which  the  appraisal  was  performed,  using 
the  ADS  template  provided  by  the  CMMI  Steward.  The  ADS  must  be 
signed  by  the  appraisal  team  leader  and  appraisal  sponsor. 


Required  elements  of  the  final  findings  include 

•  a  summary  of  the  appraisal  process 

•  the  findings  (i.e.,  summary  of  strengths  and  weaknesses) 

•  signed  verification  that  the  appraisal  team  leader  and  all  appraisal 
team  members  agree  with  the  findings  and  any  ratings  reported 

If  the  appraisal  resulted  in  capability  or  maturity  level  4  or  5  ratings,  the 
organizational  processes  or  subprocesses  and  the  corresponding  PAs  that 
were  determined  to  be  under  statistical  control  must  be  included  in  the 
final  findings. 

Appraisal  team  consensus  must  be  obtained  on  the  wording  of  the  final 
findings,  to  ensure  that  the  whole  team  supports  the  accuracy  of  the 
described  appraisal  results. 

The  team,  when  delivering  the  final  findings,  must  adhere  to  some 
important  principles: 

•  If  a  model  component  is  reported  as  Unsatisfied,  the  corresponding 
findings  of  weaknesses  that  caused  the  team  to  make  that  judgment 
must  also  be  reported. 

•  Confidentiality  and  non- attribution  principles  apply  to  statements 
made  in  the  presentation  of  final  findings. 


Con  tin  ued  on  next  page 
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3.1.1  Deliver  Final  Findings  (continued) 


Optional 

Practices 


Optional  elements  of  the  final  findings  include 

•  ratings 

•  improvement  activities 

•  recommended  actions 

•  schedule  of  major  upcoming  events  (e.g.,  appraisal  report, 
recommendations,  action  plan,  or  reappraisal) 

Note  that  the  generation  of  goal  ratings  by  the  appraisal  team  is  required 
(as  described  in  process  2.6,  Generate  Appraisal  Results).  However,  these 
ratings  may  be  excluded  from  the  final  findings  at  the  discretion  of  the 
appraisal  sponsor. 

A  formal  presentation  of  appraisal  results,  delivered  by  the  appraisal  team, 
is  frequently  the  final  visible  activity  for  appraisals  conducted  for  internal 
process  improvement.  The  final  findings  presentation  typically  is  delivered 
in  the  form  of  a  face-to-face  briefing.  Other  mechanisms  for  providing  the 
appraisal  results  to  the  organizational  unit,  such  as  written  reports,  may  be 
more  practical  in  supplier  selection  or  process  monitoring  usage  of  the 
method.  The  timeframe  in  which  the  appraisal  results  are  provided  may 
also  vary. 

The  draft  ADS  may  optionally  be  provided  during  the  executive  session(s), 
if  performed,  instead  of  at  the  conclusion  of  the  final  findings  briefing. 


Continued  on  next  page 
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3.1.1  Deliver  Final  Findings  (continued) 


Implementation 

Guidance 


The  ADS  is  a  summary  statement  describing  the  appraisal  results  that 
includes  the  conditions  and  constraints  under  which  the  appraisal  was 
performed.  It  contains  information  considered  essential  to  adequately 
interpret  the  meaning  of  assigned  maturity  level  or  capability  level  ratings. 
The  ADS  is  prepared  by  the  appraisal  team  leader  and  provided  to  the 
appraisal  sponsor. 

A  template  for  the  ADS  is  provided  in  Appendix  A.  The  ADS  is 
considered  a  draft  at  this  stage  of  the  appraisal  process,  in  that  the  ADS 
also  contains  an  affirmation  that  all  appraisal  requirements  have  been 
satisfied,  which  cannot  be  claimed  until  the  completion  of  all  appraisal 
activities. 

A  template  for  a  final  findings  briefing,  describing  its  typical  contents  and 
format,  is  provided  to  SCAMPI  Lead  Appraisers  as  a  work  aid  by  the 
CMMI  Steward. 

Findings  include  a  summary  of  strengths  and  weaknesses  determined  for 
each  PA  within  the  appraisal  reference  model  scope.  This  summary  may 
also  include  global  findings  that  apply  across  multiple  PAs,  and  non¬ 
reference  model  findings  that  affect  the  implementation  (positively  or 
negatively)  of  associated  processes  within  the  organizational  unit. 

Normally,  the  appraisal  team  leader  presents  the  final  findings.  In  some 
applications  of  the  method  for  internal  process  improvement,  the  team  may 
elect  to  have  an  appraisal  team  member  from  the  organizational  unit 
provide  the  briefing  to  encourage  the  acceptance  of  the  final  findings  and 
ownership  of  the  appraisal  results  for  follow-on  action. 

As  a  courtesy,  the  appraisal  team  can  consider  informing  the  appraisal 
sponsor  and/or  the  senior  site  manager  of  the  appraisal  results  prior  to 
presenting  them  publicly  in  the  final  findings  briefing.  This  private  briefing 
may  help  them  avoid  surprises  and  obtain  feedback  on  ways  to  present  the 
findings  that  best  meet  the  needs  of  the  sponsor,  appraisal  participants,  and 
the  organizational  unit.  See  activity  3.1.2,  Conduct  Executive  Session(s) 
for  a  description  of  topics  for  discussion. 

If  virtual  methods,  such  as  video  conferences,  teleconferences,  or  other 
similar  technology  are  used  for  the  final  findings  presentation,  the 
appraisal  team  leader  should  ensure  that  these  methods  in  no  way 
compromise  the  integrity  of  the  appraisal. 

The  number  and  scope  of  findings  reported  will  affect  the  impact  of 
appraisal  results,  whether  or  not  the  team  intends  for  it  to  happen.  There 
are  times  when  providing  a  long  list  of  details  is  beneficial.  Other  times, 
high-level  summaries  are  more  appropriate. 
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3.1.2  Conduct  Executive  Session(s) 


Activity  The  executive  session  is  an  optional  activity  that  may  be  performed  at  the 

Description  discretion  of  the  appraisal  sponsor  or  senior  site  manager.  The  executive 

session  provides  the  appraisal  sponsor,  senior  site  manager,  and  invited 
staff  a  private  opportunity  to  (a)  discuss  with  the  appraisal  team  leader  any 
issues  with  the  appraisal,  (b)  obtain  clarification  of  the  appraisal  results,  (c) 
confirm  understanding  of  the  process  issues,  and  (d)  provide  guidance 
regarding  focus,  timing,  and  priorities  of  the  recommendations  report  and 
follow-on  activities. 

Required  None.  If  the  option  is  selected,  hold  a  private  meeting  between  the 

Practices  appraisal  team  leader  and  the  sponsor,  including  any  participants  invited  by 

the  sponsor. 

Parameters  If  an  executive  session  is  conducted,  the  confidentiality  and  non-attribution 

and  Limits  of  data  sources  must  be  maintained. 

Optional  Attendance  by  the  entire  appraisal  team  at  the  executive  sessions  is  an 

Practices  option. 

The  executive  session  is  also  an  appropriate  opportunity  to  review 
appraisal  performance  with  the  appraisal  sponsor  and/or  senior  site 
manager,  and  planned  versus  actual  execution  of  the  appraisal  plan, 
including  method  tailoring.  This  session  provides  additional  input  on  the 
appropriate  expectations  for  interpreting  and  handling  the  appraisal  results. 

Multiple  sessions  may  be  held  if  necessary,  targeted  at  the  information 
needs  of  the  executive  audience. 

The  draft  ADS  may  optionally  be  provided  during  the  executive  session 
instead  of  at  the  conclusion  of  the  final  findings  briefing,  as  discussed  in 
activity  3.1.1,  Deliver  Final  Findings. 

The  intent  of  the  executive  sessions  is  to  ensure  that  the  appraisal  sponsor 
and/or  the  senior  site  manager  have  a  sound  understanding  of  the  appraisal 
results.  Any  feedback  obtained  from  these  executive  sessions  should  be 
recorded.  All  rules  for  confidentiality  and  non-attribution  are  still  in  effect. 

♦ 
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3.1 .3  Plan  for  Next  Steps 


Following  the  delivery  of  the  appraisal  results,  a  plan  for  follow-on 
activities  is  determined.  The  planned  follow-on  activities  are  typically 
defined  in  the  appraisal  plan,  reflecting  sponsor  requests  for  additional 
appraisal  tasks  and  products  necessary  to  meet  appraisal  objectives,  or  for 
a  commitment  to  take  action  on  the  appraisal  results.  Follow-on  activities 
may  include 

•  development  of  a  final  report 

•  development  of  a  recommendations  report  or  briefing 

•  generation  or  update  of  a  process  improvement  plan 

Required  None 

Practices 

Parameters  None 

and  Limits 

Optional  Planning  for  next  steps  is  an  optional,  but  recommended,  appraisal  activity. 

Practices 

Implementation  Planning  for  next  steps  includes  optional  activities  such  as 
Guidance  •  development  of  a  final  report  by  the  appraisal  team,  summarizing  the 

appraisal  results  for  delivery  to  the  appraisal  sponsor 

•  submission  of  appraisal  team  recommendations  for  action  on  the 
appraisal  findings 

•  generation  of  a  process  improvement  action  plan  for  the 
organizational  unit  to  act  on  the  appraisal  findings 

In  addition  to  specifying  the  activities  to  be  performed,  these  plans  usually 
include  the  assignment  of  responsibility,  schedule,  and  estimated  resources 
for  the  implementation  of  the  follow-on  actions.  The  plans  established  can 
be  used  to  track  the  progress  of  the  follow-on  activities  over  time. 

Findings  and  recommendations  from  the  appraisal  team  can  be  used  by  the 
organizational  unit  to  establish  action  plans  for  process  improvement.  This 
use  of  findings  and  recommendations  is  an  optional  output  most  often  used 
in  internal  process  improvement  or  process-monitoring  applications  of  the 
appraisal  method. 

Recommendations  often  include  a  prioritized  list  of  improvement 
activities,  including  the  development  of  an  improvement  plan  that  defines 
the  tasks,  schedules,  and  resources  necessary  for  implementation. 

Follow-on  appraisals  are  usually  performed  to  verify  improvement 
progress.  A  follow-on  appraisal  might  include  a  combination  of  Class  A, 
Class  B,  and  Class  C  appraisals  (refer  to  the  ARC  for  additional  details). 

Continued  on  next  page 
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3.1 .3  Plan  for  Next  Steps  (continued) 


Implementation 

Guidance 

Final  Report 


Implementation 

Guidance 

Recommend¬ 
ations  Report 


The  purpose  of  the  final  report  is  to  provide  details  or  explanations  beyond 
what  was  contained  in  the  final  findings.  The  generation  of  an  appraisal 
final  report  is  an  optional  activity  that,  if  requested  by  the  appraisal 
sponsor,  documents  the  execution  of  the  appraisal,  contains  detailed 
appraisal  findings,  and  forms  a  basis  for  action  planning.  This  baseline  is 
used  for  subsequent  reports  and  follow-on  actions,  and  also  may  be  an 
input  for  use  in  subsequent  appraisals. 

Items  contained  or  referenced  in  the  final  report,  either  in  their  entirety  or 
as  a  subset,  might  include 

•  executive  summary  of  the  appraisal  process  and  results 

•  appraisal  input  (see  process  1.1,  Analyze  Requirements) 

•  appraisal  plan  (see  process  1 .2,  Develop  Appraisal  Plan) 

•  appraisal  record  (see  process  3.2,  Package  and  Archive  Appraisal 
Assets) 

The  final  report  should  be  completed  as  soon  after  the  appraisal  as 
possible,  preferably  within  four  weeks.  The  appraisal  team  leader  usually 
generates  the  final  report;  other  team  members  may  also  contribute. 

The  format  and  content  of  the  final  report  may  vary  according  to  its 
intended  use  by  the  appraisal  sponsor.  In  its  simplest  form,  this  final  report 
could  be  a  set  of  notes  annotated  to  the  final  findings,  elaborating  on  some 
aspect  of  the  findings  or  capturing  essential  comments  or  recommendations 
from  the  appraisal  team. 


If  requested  by  the  appraisal  sponsor,  appraisal  team  recommendations  for 
taking  action  on  the  appraisal  results  can  be  provided.  These 
recommendations  can  provide  momentum  to  the  appraisal  follow-up  by 
serving  as  a  link  between  the  appraisal  findings  and  subsequent  decision 
making  or  action  plans.  The  emphasis  of  these  recommendations  depends 
on  the  appraisal  sponsor’s  objectives  and  planned  use  of  the  appraisal 
results,  as  defined  in  the  appraisal  input.  This  emphasis  can  vary  widely 
based  on  the  context  in  which  the  appraisal  method  is  applied  (i.e.,  internal 
process  improvement,  supplier  selection,  or  process  monitoring). 

The  recommendations  report  should  be  completed  as  soon  after  the 
Conduct  Appraisal  phase  as  possible.  Depending  on  the  nature, 
complexity,  and  use  of  the  recommendations,  this  report  may  take  as  long 
as  two  months  to  produce. 

Rather  than  generate  a  separate  recommendations  report,  a  common 
alternative  is  to  include  these  recommendations  in  the  final  report. 

It  is  important  to  consider  the  possibility  that  the  expertise  needed  for 
making  the  appropriate  recommendations  may  be  beyond  the  level  of 
expertise  reflected  on  the  team. 


♦ 
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3.2  Package  and  Archive  Appraisal  Assets 


Purpose 


Entry  Criteria 


Inputs 


Activities 


Outputs 


Outcome 


Exit  Criteria 


Preserve  important  data  and  records  from  the  appraisal,  and  dispose  of 
sensitive  materials  in  an  appropriate  manner. 


•  The  appraisal  has  been  conducted. 

•  Results  have  been  delivered  to  the  sponsor. 

•  All  appropriate  data  have  been  collected  and  retained  during  the 
appraisal. 


•  appraisal  data 

-  appraisal  input 

-  appraisal  plan 

-  final  findings 

-  objective  evidence 

-  signed  ADS 

•  Appraisal  team  artifacts 

-  notes 

-  documented  practice  implementation  gaps 

-  preliminary  findings 

-  document  library 


3.2. 1  Collect  Lessons  Learned 

3.2.2  Generate  Appraisal  Record 

3.2.3  Provide  Appraisal  Feedback  to  CMMI  Steward 

3.2.4  Archive  and/or  Dispose  of  Key  Artifacts 


•  appraisal  data  package 

•  appraisal  record 

•  completed  forms  and  checklists 

•  sanitized  data  (as  appropriate  and  agreed  upon  during  planning) 

•  lessons  learned  (appraisal  team,  organization) 


Data  and  artifacts  are  appropriately  archived  or  destroyed.  The  team  has 
captured  lessons  and  data  to  help  improve  the  appraisal  process. 
Requirements  for  providing  appraisal  artifacts  to  stakeholders  and  the 
CMMI  Steward  are  met. 


•  Appraisal  assets  are  baselined  and  archived. 

•  Required  reports  are  delivered  to  the  appropriate  stakeholders. 

•  Artifacts  containing  sensitive  information  are  disposed  of  in  an 
appropriate  manner. 


Con  tin  ued  on  next  page 
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3.2  Package  and  Archive  Appraisal  Assets  (continued) 


Key  Points  Protect  the  confidentiality  of  sensitive  data  while  distributing  and  archiving 

appraisal  assets.  Bundle  related  information  together  whenever 
appropriate. 

Tools  and  The  use  of  electronic  (database)  tools  for  managing  appraisal  data  often 

Techniques  provides  assistance  in  ensuring  the  integrity  of  baselines,  as  well  as 

repackaging  information  for  archival  purposes.  Electronic  tools  allow  the 
appraisal  team  leader  to  remove  traceability  information  so  that  data  can  be 
provided  to  the  appropriate  people  while  preserving  the  anonymity  of  the 
data  sources. 

Electronic  tools  also  support  the  submission  of  appraisal  data  to  the  CMMI 
Steward.  This  use  of  electronic  tools  reduces  the  administrative  burden  and 
facilitates  the  analysis  of  appraisal  method  performance  data.  These  tools 
also  provide  feedback  on  the  consolidated  analysis  results  to  the  appraisal 
community. 

Metrics  While  archiving  and  reporting  the  metrics  associated  with  the  conduct  of 

the  appraisal  is  an  important  element  of  this  activity,  the  metrics  associated 
with  the  conduct  of  this  activity  itself  are  limited.  The  effort  and  calendar 
time  consumed  are  collected  and  compared  to  the  plan.  Some  appraisal 
team  leaders  will  choose  to  maintain  personal  metrics  associated  with  the 
artifacts  described  in  this  activity. 

Verification  and  The  Required  Practices  section  of  activity  3.2.2,  Generate  Appraisal 

Validation  Record,  guides  the  verification  of  the  list  of  artifacts  provided  to  the 

sponsor. 

The  Required  Practices  section  of  activity  3.2.3,  Provide  Appraisal  Data 
Package  to  CMMI  Steward,  guides  the  verification  of  the  list  of  artifacts 
provided  to  the  CMMI  Steward  Validation  is  provided  by  the  CMMI 
Steward  upon  receipt  of  the  appraisal  data  package. 

Records  •  appraisal  record 

•  appraisal  data  package 

•  lessons  learned 

Tailoring  The  usage  mode  and  constraints  of  the  appraisal,  as  well  as  the  sensitivity 

of  the  data  and  planned  use  of  appraisal  results,  may  greatly  affect  the 
degree  to  which  appraisal  data  is  retained,  sanitized,  or  discarded. 

Con  tin  ued  on  next  page 
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3.2  Package  and  Archive  Appraisal  Assets  (continued) 


Interfaces 
with  Other 
Processes 


Summary  of 
Activities 


As  the  final  process  in  the  appraisal,  this  process  is  about  collecting, 
packaging,  and  archiving  those  results  and  artifacts  produced  by  previous 
processes  that  become  part  of  the  appraisal  record.  Most  notably,  this 
collection  of  information  includes  the  appraisal  input,  appraisal  plan,  and 
appraisal  results.  Additionally,  sensitive  or  proprietary  data  produced  by 
other  appraisal  processes  must  be  returned  to  the  organizational  unit  or 
destroyed. 


This  process  performs  the  data  collection,  data  management,  and  reporting 
activities  necessary  to  close  out  the  appraisal.  Data  collected  throughout 
the  appraisal  is  consolidated  and  baselined,  becoming  a  permanent  part  of 
the  appraisal  record. 


♦ 
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3.2.1  Collect  Lessons  Learned 


Activity  As  one  of  the  final  activities  in  wrapping  up  an  appraisal,  teams  typically 

Description  record  lessons  learned  from  their  experience.  The  purpose  of  these  lessons 

learned  is  to  document  what  went  right,  what  went  wrong,  and  any 
suggestions  or  recommendations  for  improving  the  method  or  its 
execution.  The  collection  of  lessons  learned  is  a  recommended  activity  for 
the  improvement  of  future  appraisals,  but  is  not  a  method  requirement. 

Required  None 

Practices 

Parameters  Lessons  learned  must  adhere  to  the  same  principles  of  confidentiality  and 

and  Limits  non-attribution  applicable  to  other  appraisal  results. 

All  practices  related  to  the  collection  of  lessons  learned  are  optional,  but 
recommended.  If  the  team  has  identified  potential  improvements  to 
elements  of  the  CMMI  Product  Suite  (i.e.,  appraisal  reference  model, 
appraisal  method,  and  training  materials),  these  improvements  can  be 
submitted  as  change  requests  to  the  CMMI  Steward. 

Implementation  Capturing  lessons  learned  is  often  done  as  a  group  at  the  end  of  the 
Guidance  appraisal,  while  the  appraisal  activities  are  fresh  in  team  members’  minds. 

This  collection  of  lessons  learned  can  be  supplemented  with  additional 
inputs  from  team  members  upon  further  reflection,  if  necessary.  Appraisal 
team  leaders  forward  these  aggregate  lessons  learned,  as  appropriate,  to 
various  stakeholders,  but  always  to  the  other  team  members.  Appraisal 
team  leaders  and  members  often  maintain  summary  lists  of  appraisal  best 
practices  and  lessons  learned  as  a  mechanism  for  continuous  learning  and 
improvement,  and  these  lists  are  used  as  a  resource  for  planning 
subsequent  appraisals. 

♦ 


Optional 

Practices 
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3.2.2  Generate  Appraisal  Record 


Activity 

Description 


Required 

Practices 


Appraisal  data  collected  throughout  the  appraisal  is  aggregated  and 
summarized  into  a  permanent  record  documenting  the  appraisal  conduct 
and  results.  This  collection  of  data  is  referred  to  as  the  appraisal  record  and 
is  delivered  to  the  appraisal  sponsor  for  retention. 


The  appraisal  team  leader  shall 

•  collect  and  baseline  appraisal  data  that  becomes  part  of  the  permanent 
records  provided  to  appraisal  stakeholders 

•  document  the  satisfaction  of  all  SCAMPI  A  requirements 

•  generate  the  appraisal  record  from  baselined  planning  and  execution 
data  collected  throughout  the  appraisal 

•  deliver  the  appraisal  record  to  the  appraisal  sponsor 


Continued  on  next  page 
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3.2.2  Generate  Appraisal  Record  (continued) 


Parameters 
and  Limits 


Required  contents  of  the  appraisal  record  include  the  following: 

•  dates  of  the  appraisal 

•  appraisal  input 

•  appraisal  plan 

•  objective  evidence,  or  identification  thereof,  sufficient  to  substantiate 
goal-rating  judgments 

•  characterizations  of  practice  implementation  determined  at  the 
instantiation  level  and  aggregated  at  the  organizational  unit  level 

•  identification  of  the  appraisal  method  (and  version)  used  along  with 
any  tailoring  options 

•  final  findings 

•  all  ratings  rendered  during  the  appraisal  (goals,  PAs,  and  maturity  or 
capability  levels) 

•  ADS 

Appraisal  data  must  comply  with  rules  for  non-attribution,  confidentiality, 
protection  of  proprietary  information,  and  applicable  laws,  regulations,  or 
standards  (e.g.,  acquisition  regulations  or  security  classification). 
Recipients  are  expected  to  place  the  appropriate  limitations  on  the  access 
and  use  of  the  provided  appraisal  data. 

For  high  maturity  appraisals  (capability  or  maturity  level  4  or  5),  the 
appraisal  team  leader  must  validate  that  a  substantial  portion  of  the 
organization’s  and  projects’  quality  and  process-performance  objectives 
and  statistically  managed  subprocesses  can  be  mapped  directly  to  and 
support:  (1)  the  established  business  objectives  as  stated  and  disseminated 
to  key  employees  of  the  organization,  and  (2)  the  needs  and  priorities  of 
customers,  end  users,  and  other  stakeholders.  This  validation  is  required  to 
prevent  the  granting  of  high  maturity  ratings  for  trivial  improvements. 

The  appraisal  team  leader  documents  in  the  ADS  that  all  SCAMPI  A 
requirements  were  satisfied. 


Continued  on  next  page 
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3.2.2  Generate  Appraisal  Record  (continued) 


Optional  The  appraisal  record  should  also  contain  any  additional  outputs  requested 

Practices  by  the  appraisal  sponsor,  as  agreed  on  during  appraisal  planning  and 

documented  in  the  appraisal  input. 


Implementation  The  actual  objective  evidence  (artifacts  or  portions  of  artifacts)  need  not  be 
Guidance  part  of  the  appraisal  record,  but  an  identification  of  the  objective  evidence 

is  required.  This  identification  may  be  implemented  by  providing  the  PIIs 
that  were  used  as  the  basis  for  characterizing  practice  implementation. 

Guidance  on  the  protection  of  appraisal  data  can  be  summarized  based  on 
the  recipient  of  the  data  as  follows: 

•  appraisal  sponsor:  Replacement  of  specific  sources  (persons,  projects, 
support  groups)  with  non-attributable,  general  identifiers  (e.g., 
numeric  codes  assigned  to  projects,  support  groups,  roles,  or  data- 
gathering  sessions).  If  the  sponsor  is  separate  from  the  appraised 
organization  (e.g.,  in  the  case  of  a  supplier  selection  context),  there 
may  be  situations  where  confidential  or  proprietary  data  relating  to  the 
appraised  organization  must  be  removed. 

•  CMMI  Steward:  Same  as  for  appraisal  sponsor,  for  data  that  is  shared 
by  both.  For  data  that  is  provided  only  to  the  CMMI  Steward,  the  data 
collection  vehicles  (e.g.,  forms)  are  already  designed  to  observe  non¬ 
attribution  and  confidentiality  rules.  Additionally,  supplied  data  may 
be  subject  to  further  sanitization  to  comply  with  acquisition  or 
security-related  restrictions. 

•  senior  site  manager:  In  cases  where  the  appraised  organizational  unit 
is  separate  from  the  appraisal  sponsor,  the  appraised  organization  is 
typically  provided  only  with  appraisal  results  and  not  data  related  to 
planning  and  decision  making,  or  data  that  makes  use  of  the  results. 
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3.2.3  Provide  Appraisal  Data  Package  to  CMMI  Steward 

Activity 

Description 

Appraisal  data  required  by  the  CMMI  Steward  is  collected  and  reported. 
This  data  includes  a  subset  of  the  contents  of  the  appraisal  record,  as  well 
other  data  used  by  the  Steward  to  aggregate  and  analyze  appraisal 
performance  data  for  reporting  to  the  community  and  monitoring  the 
quality  of  performed  appraisals. 

Required 

Practices 

The  appraisal  team  leader  shall 

•  submit  the  completed  appraisal  data  package  as  required  by  the 
CMMI  Steward 

The  appraisal  data  package  consists  of 

•  approved  ADS 

•  approved  appraisal  input 

•  approved  appraisal  plan 

•  the  final  findings  presentation  or  report 

Parameters 
and  Limits 

The  CMMI  Steward  defines  the  specific  set  of  data  required  for  submission 
at  the  completion  of  an  appraisal.  Submission  of  the  appraisal  data  package 
is  required  for  the  appraisal  to  be  recorded  in  the  Steward’s  database  of 
appraisal  results.  This  data  is  also  a  requirement  established  by  the  Steward 
to  maintain  SCAMPI  Lead  Appraiser  authorization. 

Optional 

Practices 

If  the  objective  evidence  is  available  in  electronic  form,  it  can  be  included 
as  part  of  the  appraisal  data  package  submitted  to  the  CMMI  Steward. 

Implementation 

Guidance 

The  appraisal  team  leader  is  responsible  for  ensuring  that  the  appraisal  data 
package  is  collected  and  reported.  The  CMMI  Steward,  as  custodian  of  the 
product  suite  and  the  Appraisal  Program,  has  several  objectives  in  seeking 
appraisal  feedback: 

•  characterization  of  the  state  of  the  practice  in  the  appraisal 
community,  for  the  collection  and  distribution  of  effective  appraisal 
techniques 

•  analysis  of  reported  appraisal  data  to  obtain  an  understanding  of 
appraisal  performance  for  continuous  improvement 

•  quality  control  within  the  Appraisal  Program,  to  ensure  a  high  level  of 
confidence  in  the  accuracy  of  appraisal  results 

The  CMMI  Steward  provides  approved  information  within  the  bounds  of 
confidentiality  to  the  community,  based  on  results  from  the  appraisal  data 
collected.  The  Steward  establishes  the  format  and  mechanisms  for  the 
presentation  of  this  information. 

♦ 
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3.2.4  Archive  and/or  Dispose  of  Key  Artifacts 


Activity 

Description 


Required 

Practices 


Parameters 
and  Limits 


Optional 

Practices 


After  the  various  reports  are  delivered  to  the  appropriate  stakeholders  and 
the  appraisal  assets  have  been  baselined,  the  appraisal  team  leader  is 
responsible  for  properly  archiving  and/or  disposing  of  the  appraisal  data,  in 
accordance  with  agreements  made  with  the  sponsor  and  documented  in  the 
appraisal  input.  The  team  librarian  (if  one  is  used)  ensures  that  all 
organization-provided  documentation  and  objective  evidence  is  returned  or 
disposed  of  properly.  Any  remaining  team  artifacts  or  notes  are  disposed  of 
properly. 


The  appraisal  team  leader  shall 

•  archive  or  dispose  of  key  artifacts  collected  by  the  appraisal  team 

•  return  objective  evidence  provided  by  the  organizational  unit 


In  all  usage  modes  of  SCAMPI  A,  strict  non-attribution  policies  apply. 
Confidentiality  and  non-disclosure  agreements  established  with  the 
appraisal  team  members  remain  in  effect. 


None 


Con  tin  ued  on  next  page 


CMU/SEI-2006-HB-002 


Page  11-133 


3.2.4  Archive  and/or  Dispose  of  Key  Artifacts  (continued) 


Implementation 

Guidance 


How  the  records  will  be  preserved  or  disposed  of  is  dependent  on  the  usage 
mode  of  the  method  and  the  appraisal  objectives  that  shape  the  current 
application.  Confidentiality  rules  may  differ  by  application.  In  a  supplier 
selection  usage,  the  results  are  not  proprietary  in  that  the  sponsor  is  not  a 
member  of  the  appraised  organization.  However,  results  are  only  known  to 
the  sponsor  and  the  recipient;  competing  organizations  do  not  see  the 
results.  Confidentiality  of  results  can  be  characterized  as  one  of  the 
following: 

•  known  only  to  the  recipient  organization 

•  known  to  the  recipient  and  sponsor,  when  they  are  from  different 
organizations 

•  known  to  anyone 

The  sponsor  is  solely  responsible  for  determining  the  confidentiality  with 
which  the  appraisal  results  will  be  maintained.  The  non-attribution  of  data 
to  specific  individuals  is  the  responsibility  of  the  appraisal  team.  The 
recipient  organization,  if  the  sponsor  agrees  and  it  is  planned  for,  may 
always  choose  to  make  the  results  known  outside  the  organization.  At  a 
high  level,  this  disclosure  might  be  done  for  marketing  and  public  relations 
reasons.  Disclosures  of  results  include  the  context  and  constraints  under 
which  the  appraisal  was  performed  (e.g.,  appraisal  reference  model  scope 
and  organizational  scope),  as  defined  by  the  ADS  described  in  process  3.1, 
Deliver  Appraisal  Results. 

Any  annotations  related  to  the  objective  evidence  provided  to  the 
organization  by  the  appraisal  team  should  be  recorded  and  archived  for  use 
in  process  improvement  actions  or  for  reuse  in  subsequent  appraisals. 
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Appendix  A  SCAMPI  A  Appraisal 

Disclosure  Statement  (ADS) 


Example  Template 


Section  1:  Appraisal  Identification  provides  for  the  disclosure  of  information,  which  meets 
the  minimum  requirements  for  the  appraisal  disclosure  statement  as  defined  for  the  SCAMPI 
appraisal  method. 1 

Section  2:  Affirmations  provides  for  the  affirmation  of  the  accuracy  of  this  ADS  by  the  SEI- 
authorized  SCAMPI  Lead  Appraiser  and  the  sponsor.  These  affirmations  are  required.  There 
are  also  options  that  can  be  initialed  by  the  sponsor  to  request  and  provide  authorization  for 
information  in  this  ADS  to  be  published  on  Web  sites. 


1  Appendix  A,  Standard  CMMI  Appraisal  Method  for  Process  Improvement  (SCAMPI),  Version 
1.2:  Method  Definition  Document,  SEI  Handbook  CMU/SEI-2006-HB-002,  August  2006. 
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Section  1:  Appraisal  Identification 

Organization  Name 

Provide  the  name  of  the  organization.  The  organization  is  the  legal  entity  for  which  this 
appraisal  is  being  conducted. 

Organizational  Unit 

Provide  a  description  of  the  organizational  unit.  The  organizational  unit  is  that  part  of  an 
organization  that  is  the  subject  of  an  appraisal  and  to  which  the  appraisal  results  will  be 
generalized.  It  is  possible  that  the  organization  and  the  organizational  unit  may  be  one  and  the 
same. 

Projects  within  the  Organizational  Unit  Excluded  from  this  Appraisal 

List  projects,  categories  of  projects,  or  groups/functions  that  were  specifically  excluded  from 
this  appraisal  and  a  justification  for  their  exclusion.  Note:  The  appraisal  results  (e.g.,  findings 
and  ratings)  do  not  apply  to  these  projects  and  groups/ftmctions. 

Organizational  Scope 

Provide  a  description  of  the  organizational  scope  for  this  appraisal.  The  organizational  scope 
is  the  portion  of  the  organizational  unit  that  participates  in  the  appraisal.  It  is  possible  that  the 
organizational  unit  and  organizational  scope  may  be  one  and  the  same. 

Organizational  Projects  and  Support  Groups 

Specify  all  of  the  projects  and  support  groups  that  are  included  within  the  scope  of  this 
appraisal  and  from  which  objective  evidence  has  been  obtained  and  analyzed.  Together,  these 
projects  and  support  groups  should  provide  examples  of  practices  used  in  appropriate 
contexts  within  the  boundaries  of  the  organizational  unit.  In  addition,  provide  some  rationale 
for  the  projects  and/or  support  groups  that  comprise  the  organizational  scope  of  this 
appraisal. 


Project  or  Support  Group  1 

Project  or  support  group  name: 


Project  type  (Enter  Y  or  N.) 


Focus  project?  (yes  or  no) 

Non-focus  project?  (yes  or  no) 

Support  group?  (yes  or  no) 
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Process  areas  for  which  objective  evidence  was  collected  and  analyzed: 


Description  of  the  project  or  support  group: 


Size  (i.e.,  number  of  people): 


Organizational  function  (e.g.,  development  project,  maintenance  group,  or  CM  support 
group): 


Placement  within  organizational  unit  (i.e.,  affiliation  with  a  particular  division,  product  line, 
or  management  chain): 


Geographic  location(s)  of  project  or  support  group  (i.e.,  country,  state,  city): 


Project  start  date  and  (projected)  end  date: 


Sponsor  initial  if  this  appraisal  is  for  a  sensitive  or  proprietary  project  or  support 
group  and  is  not  to  be  publicly  disclosed. 

Note:  A  point  of  contact  (POC)  for  additional  information  on  this  sensitive  or 
proprietary  project  or  support  group  is  required: 

POC  Name _ 

Phone  Email 


Project  or  Support  Group  N 

Use  the  same  descriptor  as  listed  above  for  each  project  or  support  group  within  the 
organizational  scope. 
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Projects  or  Support  Groups  Providing  Additional  Objective  Evidence 

List  remaining  projects  or  groups  that  provided  additional  objective  evidence  for  this 
appraisal.  These  projects  or  groups  may  or  may  not  have  been  specified  in  the  appraisal  input 
and  plan,  depending  on  the  actual  execution  of  the  appraisal  and  need  for  additional  objective 
evidence.  Use  the  same  descriptor  items  as  listed  above. 

•  Project  or  support  group  1 

•  Project  or  support  group  N 

Organizational  Sample  Size 

Define  the  percentages  of  people,  projects,  and  support  groups  included  in  the  organizational 
scope  in  relation  to  the  organizational  unit. 


%  of  people  included:  #  of  people  in  the  organizational  scope  divided  by  #  of 
people  in  the  organizational  unit  (xlOO) 

%  of  projects  included  :  #  of  projects  in  the  organizational  scope  divided  by  #  of 
projects  in  the  organizational  unit  (xlOO) 

Critical  Factors 

Identify  factors  critical  to  the  implementation  of  practices  in  projects.  Provide  the  percentage 
of  each  critical  factor  identified  in  appraisal  planning  covered  by  the  organizational  scope  in 
relation  to  the  organizational  unit.  Examples  of  critical  factors  include  the  following: 

•  application  domains  (or  lines  of  business) 

•  geographical  breadth 

•  disciplines  (e.g.,  systems  engineering,  software  engineering,  and  hardware  engineering) 

•  effort  types  (e.g.,  development,  maintenance,  and  services) 

•  project  types  (e.g.,  legacy  and  new  development) 

•  customer  type  (e.g.,  commercial,  DoD,  and  NASA) 

•  lifecycle  models  in  use  within  the  organization  (e.g.,  spiral,  evolutionary,  waterfall,  and 
incremental) 
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Appraisal  Schedule 


Start  Date 

Completion  Date 

Activity/Event 

Remarks 

Phase  1  (Plan  and  Prepare  for 
Appraisal) 

In  this  column,  include  any 

information  to  add  context  for 

atypical  schedules  and  milestones  as 

necessary. 

Phase  2  (Conduct  Appraisal) 

Phase  3  (Report  Results) 

Appraisal  Expiration  Date 

The  findings  and  results  from  a  SC  AMPI  A  Appraisal  are  valid  for  a  period  of  not  more  than 
three  years  from  delivery  of  the  appraisal  findings.  Regardless  of  the  period  of  validity  of  the 
findings  and  results,  there  are  several  factors  that  may  impact  the  validity  or  accuracy  of  the 
data  at  the  time  of  inquiry.  Factors  such  as  significant  reorganizations,  mergers,  significant 
changes  in  senior  leadership,  significant  workforce  turnover,  and  erosion  of  sponsorship  may 
need  to  be  considered  by  the  recipient  of  an  ADS  or  in  the  summary  of  the  ADS  information. 

For  this  appraisal  the  expiration  date  is: 


Appraisal  Method  Used 

Name  (Choose  one  with  an  X  in  the  corresponding  cell): 


SCAMPI  A  Version  1.1 

SCAMPI  A  Version  1.2 

If  other  method  used,  please  describe: 

Representation  (Choose  one  with  an  X  in  the  corresponding  cell): 


CMMI  Staged  V  1.1 

CMMI  Continuous  V 1 . 1 

CMMI  Staged  VI. 2 

CMMI  Continuous  V 1 .2 
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Appraised  functional  areas  (examples) 


Software 

Systems  Engineering 

Hardware 

Program  Management 

Supply  Chain  Management 

Logistics 

Other  (Specify) 

For  SCAMPI-A  Version  1 .2,  indicate  constellation  used 


DEV 

DEV+IPPD 

ACQ 

SVC 

CMMI  Model  Scope  Description 

Place  a  “Y”  in  the  “In  Scope”  column  for  each  process  area  within  scope  of  the  appraisal. 
Place  an  “N”  in  that  column  for  each  process  area  not  within  scope  of  the  appraisal.  Place  an 
“NA”  in  each  column  for  each  process  area  deemed  to  be  not  applicable  for  the 
organizational  unit.  Provide  rationale  for  process  areas  deemed  not  applicable.  (Note:  For 
CMMI -DEV,  vl.2,  Supplier  Agreement  Management  [SAM]  is  the  only  process  area  that 
may  be  designated  NA.) 


In  Scope 

Category 

Process  Area 

Process  Management 

Organizational  Process  Focus 

Organizational  Process  Definition  (V  1.1) 

Organizational  Process  Definition  (Vl.2;  1  additional 
Goal  for  IPPD) 

Organizational  Training 

Organizational  Process  Performance 

Organizational  Innovation  and  Deployment 

Project  Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 
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Project  Management 

(continued) 

Integrated  Project  Management  (VI.  1;  2  additional 

Goals  for  IPPD) 

Integrated  Project  Management  (VI. 2;  1  additional  Goal 
for  IPPD) 

Risk  Management 

Integrated  Teaming  (Vl.l  only;  IPPD) 

Integrated  Supplier  Management  (Vl.l  only;  SS) 

Quantitative  Project  Management 

Engineering 

Requirements  Management 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Support 

Configuration  Management 

Process  and  Product  Quality  Assurance 

Measurement  and  Analysis 

Decision  Analysis  and  Resolution 

Organizational  Environment  for  Integration  (Vl.l  only; 
IPPD) 

Causal  Analysis  and  Resolution 
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Key  Appraisal  Participants 


Appraisal  Role 

Name 

Organizational 

Affiliation 

Appraisal  sponsor 

Appraisal  team  leader 

Appraisal  team  member 

Appraisal  team  member 

Appraisal  team  member 

Appraisal  team  member 

Appraisal  team  member 

Appraisal  team  member 

Appraisal  team  member 

Appraisal  team  member 

Appraisal  Ratings 

Capability  Level  Rating(s)  (for  Continuous  Representation) 

Unless  otherwise  specified,  the  ratings  indicated  below  were  assigned  to  the  organizational 
unit  indicated  above. 

For  each  goal,  please  designate  one  of  the  following: 

S:  Satisfied  U:  Unsatisfied 

NR:  Not  Rated 

For  each  process  area  (PA)  within  scope  of  the  appraisal,  indicate 

•  the  capability  level  (CL)  in  the  range  of  CL0-CL5  or 

•  NA  if  the  PA  is  not  applicable  in  the  organizational  unit  (i.e.,  outside  of  its 
scope  of  work)  or 

•  NR  if  any  of  the  goals  associated  with  the  PA  are  not  rated  or 

•  OS  if  the  PA  is  out  of  scope  for  this  appraisal. 


PA 

Name 

SGI 

SG2 

SG3 

SG4 

GG1 

GG2 

GG3 

GG4 

GG5 

PA 

Rating 

OPF 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

OPD 
(VI. 1) 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

OPD 
(VI  .2) 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 
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PA 

Name 

SGI 

SG2 

SG3 

SG4 

GG1 

GG2 

GG3 

GG4 

GG5 

PA 

Rating 

OT 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

OPP 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

OID 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

PP 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

PMC 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

SAM 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

IPM 
(VI. 1) 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

IPM 
(VI  .2) 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

RSKM 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

IT 

(Vl.l) 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

ISM 

(Vl.l) 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

QPM 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

REQM 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

RD 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

TS 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

PI 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

VER 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

VAL 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

CM 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

PPQA 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

MA 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

DAR 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

OEI 

(Vl.l) 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

CAR 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

CLn 

Organizational  unit  maturity  level  rating  if  derived  through  equivalent  staging:  maturity 
level  X (1-5) 


CMU/SEI-2006-HB-002 


111-11 


Maturity  Level  Rating(s):  Staged  Representation 

Unless  otherwise  specified,  the  rating  (s)  indicated  below  were  assigned  for  the 
organizational  unit  indicated  above. 

Organizational  unit  maturity  level  rating  (if  reported):  maturity  level  X  (1-5) 

Goal  and  Process  Area  Ratings 

For  each  goal,  please  designate  one  of  the  following: 

S:  Satisfied  U:  Unsatisfied 

NR:  Not  rated 

For  each  PA  please  designate  one  of  the  following: 

•  S:  Satisfied 

•  U:  Unsatisfied 

•  NA  if  the  PA  is  not  applicable  in  the  organizational  unit  (i.e.,  outside  of  its 
scope  of  work) 

•  NR  if  any  of  the  goals  associated  with  the  PA  are  not  rated 

•  OS  if  the  PA  is  out  of  scope  for  this  appraisal. 


PA  Name 

SGI 

SG2 

SG3 

GG2 

GG3 

PA  Rating 

REQM 

Rating 

Rating 

Rating 

Rating 

PP 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

PMC 

Rating 

Rating 

Rating 

Rating 

Rating 

PPQA 

Rating 

Rating 

Rating 

Rating 

Rating 

MA 

Rating 

Rating 

Rating 

Rating 

Rating 

CM 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

SAM 

Rating 

Rating 

Rating 

Rating 

Rating 

PA  Name 

SGI 

SG2 

SG3 

SG4 

GG2 

GG3 

PA  Rating 

RD 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

TS 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

PI 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

VER 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

VAL 

Rating 

Rating 

Rating 

Rating 

Rating 

OPF  (VI. 1) 

Rating 

Rating 

Rating 

Rating 

Rating 

OPF  (VI. 2) 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

OPD  (VI. 1) 

Rating 

Rating 

Rating 

Rating 

OPD  (VI. 2) 

Rating 

Rating 

Rating 

Rating 

Rating 
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PA  Name 

SGI 

SG2 

SG3 

SG4 

GG2 

GG3 

PA  Rating 

OT 

Rating 

Rating 

Rating 

Rating 

Rating 

IPM  (VI. 1) 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

IPM  (VI. 2) 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

RSKM 

Rating 

Rating 

Rating 

Rating 

Rating 

Rating 

IT  (VI. 1) 

Rating 

Rating 

Rating 

Rating 

Rating 

ISM  (VI. 1) 

Rating 

Rating 

Rating 

Rating 

Rating 

DAR 

Rating 

Rating 

Rating 

Rating 

OEI  (VI. 1) 

Rating 

Rating 

Rating 

Rating 

Rating 

PA  Name 

SGI 

SG2 

SG3 

GG2 

GG3 

PA  Rating 

OPP 

Rating 

Rating 

Rating 

Rating 

QPM 

Rating 

Rating 

Rating 

Rating 

Rating 

PA  Name 

SGI 

SG2 

SG3 

GG2 

GG3 

PA  Rating 

OID 

Rating 

Rating 

Rating 

Rating 

Rating 

CAR 

Rating 

Rating 

Rating 

Rating 

Rating 

Additional  Information  for  Appraisals  Resulting  in  Capability 
or  Maturity  Level  4  or  5  Ratings 

Describe  which  processes  or  subprocesses  are  under  statistical  management  and  were 
included  in  the  objective  evidence  for  this  appraisal.  Also  list  the  PA(s)  and  organizational 
quality  and  process-performance  objective(s)  these  processes  or  subprocesses  pertain  to. 


Process/Subprocess 

Process  Area(s) 

Quality /Process  Performance  Objective(s) 
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Section  2:  Affirmations 


Appraisal  Team  Leader  (required) 

I  affirm  that  the  information  in  this  statement  is  accurate  and  that  the  appraisal  described 
herein  was  conducted  in  full  accordance  with  the  requirements  of  the  SCAMPI  A  appraisal 
method  and  the  provisions  of  my  authorization  as  a  SCAMPI  Lead  Appraiser.  I  also  verify 
that,  to  the  best  of  my  knowledge,  the  organizational  scope  as  depicted  herein  is 
representative  of  the  organizational  unit  for  this  appraisal. 

For  High  Maturity  Appraisals  (Capability  or  Maturity  Level  4  or  5) 

I  validate  that  a  substantial  portion  of  the  organization’s  and  projects’  quality  and  process 
performance  objectives  and  statistically  managed  subprocesses  can  be  mapped  directly  to  and 
support: 

•  the  established  business  objectives  as  stated  and  disseminated  to  key  employees  of 
the  organization,  and 

•  the  needs  and  priorities  of  customers,  end  users,  and  other  stakeholders. 

This  validation  is  required  to  prevent  the  granting  of  high  maturity  ratings  for  trivial 
improvements. 


Appraisal  team  leader  name: _ 

Appraisal  team  leader  signature: 
Date  signed: _ 


Appraisal  Sponsor  (required) 

I  affirm  that  the  organizational  scope  described  above  is  representative  of  the  organizational 
unit  for  this  appraisal. 

I  further  affirm  that  the  information  in  this  statement  is  accurate  and  that  the  SEI  may  review 
the  appraisal  record  and/or  conduct  interviews  deemed  necessary  upon  request.  See  Activity 
3.2.2  for  contents  of  the  appraisal  record. 

I  agree  to  maintain  the  appraisal  record  through  the  date  of  expiration  of  this  appraisal  as 
documented  in  this  Appraisal  Disclosure  Statement.  (See  activity  3.2.2  of  the  SCAMPI 
Method  Definition  Document  [MDD]  for  the  content  of  the  appraisal  record.) 
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I  understand  that,  for  the  period  of  validity  of  this  appraisal,  it  is  my  responsibility,  with 
consideration  of  non- attribution  (see  MDD  activity  3.2.2),  to  ensure  that  the  appraisal  record 
is  stored  under  appropriate  levels  of  control  to  support  reviews  and/or  audits  by  the  CMMI 
Steward. 


Appraisal  sponsor  name: _ 

Appraisal  sponsor  signature: 
Date  signed:  _ 


Initial  here  if  you  are  authorizing  and  requesting  that  the  information  from  this 
ADS  be  published  on  the  Published  Appraisal  Results  System  (PARS),  a  Web  site 
providing  public  notice  of  appraisal  results. 


Initial  here  if  you  are  authorizing  and  requesting  that  the  information  from  this 
ADS  be  published  on  the  U.S.  Department  of  Defense  Web  site,  a  government  site 
providing  public  notice  of  appraisal  results. 

Note:  It  is  mandatory  for  all  organizations  for  which  the  appraised  scope  is 
primarily  US  defense  to  publish  their  results  on  that  site. 


CMU/SEI-2006-HB-002 


111-15 


111-16 


CMU/SEI-2006-HB-002 


Appendix  B  The  Role  of  Practice 

Implementation  Indicators  in 
Verifying  Practice 
Implementation 


Purpose 

This  appendix  provides  a  conceptual  overview  of  the  process  of  verifying  practice 
implementation  and  the  role  of  Practice  Implementation  Indicators  (PIIs)  in  that  process. 
Verification  of  practice  implementation  is  an  essential  element  of  appraising  the 
implementation  of  processes  relative  to  models  of  best  practices  such  as  CMMI. 

Verifying  Practice  Implementation 

In  this  discussion,  verifying  CMMI  practice  implementation  means  the  substantiation  of 
practice  implementation  based  on  a  review  of  objective  evidence.  For  example,  one  might 
inquire  as  to  whether  a  project-specific  practice  is  implemented  within  a  project. 
Alternatively,  one  might  inquire  as  to  whether  an  organization-specific  practice  is 
implemented  within  an  organization. 

Having  a  well-defined  approach  for  verifying  practice  implementation  is  of  critical 
importance  from  several  perspectives.  For  the  process  improvement  sponsor,  it  provides 
some  assurance  that  the  resources  applied  to  the  improvement  effort  will  result  in  the  desired 
outcome  and  that  the  resultant  benefits  can  therefore  be  expected.  For  process  improvement 
agents  or  champions,  it  enables  them  to  know  when  they  have  succeeded  with  the 
implementation  activity,  and  to  informally  monitor  whether  the  practice  continues  to  be 
implemented  over  time.  For  appraisal  teams,  a  well-defined  verification  approach  is  essential 
for  determining  what  capability  level  or  maturity  level  ratings  are  warranted.  CMMI  process 
area  goal  satisfaction  is  predicated  on  implementation  of  the  relevant  specific  or  generic 
practices  (or  acceptable  alternatives)2.  Hence  verification  of  practice  implementation  is  a 
crucial  appraisal  task. 


2  See  “Required,  Expected,  and  Informative  Components”  in  Chapter  2  of  the  CMMI  model  that 
you  are  using. 
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Practice  Implementation  Indicators 

The  fundamental  idea  of  PIIs  is  quite  simple  and  broadly  applicable  to  any  practice  or 
activity.  It  is  based  on  the  obvious  fact  that  the  conduct  of  an  activity  or  the  implementation 
of  a  practice  will  result  in  “footprints” — evidence  that  the  activity  was  conducted  or  the 
practice  was  implemented. 

For  example,  if  one  balances  one’s  checkbook  at  the  end  of  the  month,  there  are  several 
potential  ways  to  confirm  that  this  activity  has  indeed  taken  place.  First,  the  person  who 
engaged  in  the  checkbook  balancing  activity  can  affirm  that  this  activity  was  conducted. 
Second,  there  will  likely  be  an  entry  in  the  checkbook  register  for  each  check  or  transaction 
to  indicate  that  it  matches  with  a  corresponding  entry  in  the  hank’s  statement.  Additional 
artifacts  could  be  identified. 

The  general  idea  is  clear:  the  actual  conduct  of  an  activity  leaves  footprints  that  provide  a 
basis  for  verification. 

PIIs  refer  to  the  footprints  that  are  the  necessary  and  unavoidable  consequence  of  practice 
implementation.  They  include  information  contained  in  artifacts  and  information  gathered 
from  interviews  with  managers  and  practitioners. 

The  Role  of  PIIs 

ARC-compliant  appraisal  methods  employ  one  or  more  types  of  objective  evidence.  An 
appraisal  team  bases  its  decisions  about  practice  implementation  on  examination  of  this 
objective  evidence. 

Once  a  project  or  organization  has  an  understanding  of  how  its  processes  relate  to  the  CMMI 
model,  the  stage  is  set  for  capturing  the  PIIs  that  provide  the  objective  evidence  of 
implementation.  The  work  of  establishing  the  collection  of  PIIs  for  projects  and  the 
organization  provides  assurance  to  the  process  improvement  sponsor  that  the  expected 
implementation  activities  have  in  fact  resulted  in  alignment  of  the  organization’s  activities  with 
the  CMMI  model. 

This  aggregation  of  objective  evidence — the  PIIs — is  itself  an  important  organizational 
process  asset  that  has  a  number  of  potential  uses,  most  notably  providing  an  appraisal  team  a 
head  start  in  understanding  the  organization’s  implementation  of  the  CMMI  model.  This 
approach  leaves  the  appraisal  team  the  task  of  verifying  whether  the  objective  evidence3 
provided  is  adequate  for  substantiation  of  practice  implementation,  rather  than  the  more 
difficult,  error  prone,  and  time-consuming  task  of  investigating  each  practice  to  discover  the 
objective  evidence  needed  to  substantiate  implementation. 


3  The  ARC  defines  objective  evidence  as  “qualitative  or  quantitative  information,  records,  or  state¬ 
ments  of  fact  pertaining  to  the  characteristics  of  an  item  or  service  or  to  the  existence  and  imple¬ 
mentation  of  a  process  element,  which  are  based  on  observation,  measurement,  or  test  and  are 
verifiable.” 
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Both  the  appraised  organization  and  the  appraisal  team  have  a  clearer  picture  of  what  artifacts 
must  be  provided  to  substantiate  implementation  of  the  practices,  thereby  minimizing  the 
amount  of  further  investigation  necessary  in  the  form  of  interviews  and  additional 
documentation  requests.  The  extent  to  which  the  appraised  organization  can  provide  this 
information  becomes  a  principal  factor  in  how  much  further  investigation  may  be  required. 

Another  benefit  of  this  approach  is  significantly  greater  reliability  and  accuracy  of  appraisal. 

The  Pll-based  approach  is  not  meant  to  turn  the  appraisal  into  a  documentation  review 
exercise.  It  merely  allows  for  more  focused  and  effective  use  of  the  on-site  phase  and 
potentially  a  shorter  on-site  phase  than  would  otherwise  be  the  case. 

Finally,  the  PIIs  are  not  intended  to  tie  the  hands  of  model  implementers  or  process  appraisal 
teams.  The  primary  value  of  the  PIIs  lies  in  making  explicit  what  has  heretofore  been  implicit 
and  therefore  subject  to  wide  variations  in  interpretation  and  understanding.  Over  time, 
sharing  of  PIIs  will  result  in  a  set  of  practice  implementation  scenarios  (e.g.,  small,  medium, 
and  large  organizations  or  projects)  and  a  standard  set  of  PIIs  that  could  be  used  as  a  starting 
point  for  further  customization.  The  particular  process  implementation  context  and  the 
specifics  of  the  instantiation  would  determine  which  of  the  indicators  make  sense  for  that 
implementation.  Appraisal  teams  would  be  obliged  to  inquire  into  the  existence  of  the  agreed- 
upon  indicators,  while  still  having  the  freedom  to  make  judgments  based  on  the  facts  and 
circumstances  of  the  implementation. 

A  standard  set  of  PIIs  could  establish  norms  within  which  most  implementations  will  fall, 
thereby  allowing  efficiencies  to  be  realized  in  implementation  and  appraisal,  while  at  the 
same  time  recognizing  that  alternative  implementations  may  be  possible  using  alternative 
practices. 

Pll  Components 

PIIs  have  two  components  or  dimensions:  an  objective  evidence  component  and  a  practice 
implementation  type  component.  The  objective  evidence  component  refers  to  the  form  of  the 
objective  evidence.  The  practice  implementation  type  component  deals  with  the  significance 
of  the  objective  evidence  in  relation  to  practice  implementation. 
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Forms  of  Objective  Evidence 

An  appraisal  team  bases  its  decisions  about  practice  implementation  on  the  existence  of 
objective  evidence  available  to  it.  This  objective  evidence  can  take  on  one  or  more  of  the 
following  forms: 

•  artifacts 

-  work  products,  which  are  the  explicit  intended  consequences  of  practice  implementation 

-  artifacts  that  are  incidental  to,  but  indicative  of,  practice  implementation 

•  affirmations 

-  written  or  oral  statements  indicative  of  practice  implementation  from  practitioners 
who  carry  out  the  activities  relevant  to  the  practice  or  from  suppliers,  customers,  or 
other  stakeholders  in  the  practice 

-  demonstrations  or  presentations  (e.g.,  the  demonstration  of  capability  of  a  tool  or 
other  mechanism  as  it  relates  to  the  implementation  of  a  practice,  or  a  presentation 
explaining  some  aspect  of  the  organization  or  project) 


Note  that  there  is  not  a  strong  distinction  made  in  the  model  between  artifacts  and  work 
products.  As  used  in  the  context  of  the  CMMI  models,  “work  product”  refers  to  an  artifact 
that  is  either  explicitly  mentioned  in  the  statement  of  a  CMMI  practice  or  whose  absence 
would  be  a  strong  indictor  of  incomplete  or  inadequate  practice  implementation.  The  term 
“artifact”  is  used  in  the  context  of  the  CMMI  models  to  refer  to  an  artifact  whose  existence  is 
incidental  to  (i.e.,  a  side-effect  of)  the  accomplishment  of  the  main  intent  of  the  practice. 

Types  of  Plls 

Using  the  above  discussion  as  the  framework,  it  is  now  possible  to  itemize  the  types  of  PIIs 
that  might  be  present  as  a  consequence  of  practice  implementation.  Table  III- 1  shows  PII 
types,  which  collectively  provide  coverage  for  any  CMMI  practice.  Each  type  is  described  in 
more  detail  below. 


Table  1 1 1-1:  PII  Types 


PII  Type 

Generic  Description 

Direct  artifact 

The  tangible  outputs  resulting  from  implementation  of  a 
specific  or  generic  practice. 

Indirect  artifact 

An  artifact  that  is  a  consequence  of  performing  a  specific  or 
generic  practice  or  that  substantiates  its  implementation,  but 
which  is  not  the  purpose  for  which  the  practiced  is 
performed. 

Affirmation 

An  oral  or  written  statement  confirming  or  supporting 
implementation  (or  lack  of  implementation)  of  a  CMMI 
model  specific  practice  or  generic  practice. 
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Direct  Artifact 


This  PII  type  is  relevant  when  establishment  of  a  work  product  is  an  integral  part  of  practice 
implementation.  Sometimes  this  connection  to  a  practice  is  explicit,  as  in  “Establish  and 
maintain  process  action  plans  to  address  improvements  to  the  organization’s  processes  and 
related  process  assets”  (OPF  SP  2.1-1).  In  other  instances,  it  is  not  explicit,  although  it  would 
be  difficult  to  imagine  practice  implementation  without  the  presence  of  one  or  more  work 
products  being  produced.  In  most  cases,  the  model  document  already  identifies  these  work 
products. 

indirect  Artifact 

This  PII  type  applies  to  artifacts  that  are  produced  as  a  natural  consequence  of  practice 
enactment.  The  difference  between  an  indirect  and  a  direct  artifact  PII  is  that  this  type  applies 
to  artifacts  that  are  an  indirect  consequence  or  side-effect  of  practice  enactment.  For  this 
reason,  artifacts  that  are  relevant  to  this  PII  will  vary  widely  and  will  tend  to  be 
implementation  specific.  This  indicator  type  is  especially  useful  when  there  may  be  doubts 
about  whether  the  intent  of  the  practice  has  been  met  (e.g.,  a  work  product  exists  but  there  is 
no  indication  of  where  it  came  from  or  who  developed  it). 

Affirmation 

This  PII  type  refers  to  either  oral  or  written  statements  provided  by  individuals  involved  in 
the  enactment  of  a  practice  or  by  individuals  who  are  stakeholders  (e.g.,  customers  and 
suppliers)  in  the  enactment  of  a  practice.  This  type  can  also  apply  to  information  provided  in 
other  ways,  such  as  demonstrations  and  presentations.  While  both  oral  and  written 
affirmations  can  be  used  to  confirm  practice  implementation,  only  oral  affirmations  can  be 
used  to  meet  the  coverage  criteria  defined  in  activity  2.4.1. 

PII  Descriptions 

A  PII  Description  (PHD)  is  a  structure  or  schema  defined  to  provide  a  repository  for  the  PII 
information.  Table  III-2  shows  an  example  of  such  a  structure.  Note  that  this  example  is  a 
notional  description  of  the  content,  not  a  physical  definition  of  the  format. 


Table  111-2:  A  PHD  Schema 


Attribute 

Synopsis 

Remarks 

Practice  ID 

This  ID  identifies  the  process  area,  goal,  and  practice 
that  the  PII  is  associated  with. 

Acronyms  are  found  in 
the  CMMI  models. 

PII  ID 

This  ID  identifies  the  indicator  type  and  the  form  of 
objective  evidence. 

Types  are  direct  artifact, 
indirect  artifact,  and 
affirmation. 

Description 

This  attribute  is  a  description  of  the  PII  as  applied  to 
this  practice. 
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Attribute 

Synopsis 

Remarks 

Examples 

These  attributes  are  examples  of  artifacts  or 
affirmations  that  would  exemplify  the  intent  of  the 

PII  and/or  exploratory  questions  (EQs)  or  “look  fors” 
(LFs).  They  assist  appraisers  in  identifying  relevant 
artifacts  or  eliciting  relevant  information. 

Aim  to  minimize  any 
overlap  with  such 
information  that  is 
already  in  the  model 
document. 

Organizational 

Implementation 

This  attribute  would  be  filled  in  by  the  organization 
as  part  of  its  implementation  program  and  provided 
to  the  appraisal  team  as  a  resource. 

Table  III-3  shows  an  example  PHD  for  specific  practice  1.1-1  of  the  Project  Planning  process 
area: 


Table  111-3:  An  Example  PHD 


Attribute 

Value 

Practice  ID 

PP  SP  1.1-1 

PII  ID 

Direct  artifact 

PII  Description 

Work  product(s)  that  reflect  (document  the  information  content  of)  the 
establishment  of  a  top-level  work  breakdown  structure  (WBS)  to  estimate  of  the 
scope  of  the  project. 

Examples 

See  typical  work  products. 

Organizational 

Implementation 

{To  be  provided  by  the  organization  for  a  specific  implementation. } 

These  descriptions  have  a  number  of  uses  in  addition  to  their  utility  during  process  appraisal. 
They  can  be  used  during  the  model  implementation  phase,  after  model  implementation  as  a 
training  vehicle  for  new  personnel,  for  internal  monitoring  of  practice  implementation,  etc. 

Application  of  Plls  in  Model  Implementation 

The  use  of  indicators  has  significant  utility  for  an  organization  that  is  committed  to  model- 
based  process  improvement.  Typically,  organizations  will  either  implement  model  practices 
directly  or  will  ensure  that  the  practices  used  in  the  organization  affect  goal  achievement 
(through  the  mechanism  of  alternative  practices). 

Since  models  are  necessarily  expressed  in  an  implementation-independent  manner,  the 
implementation  of  a  model  will  require  that  an  understanding  of  how  the  model  intent  (as 
expressed  though  goals,  practices,  and  other  model  material)  is  to  be  realized  in  the 
organization  be  developed,  documented,  and  operationalized.  The  model  intent  is  made  real 
through  its  impact  on  the  way  people  work;  if  there  is  no  relation  between  how  they  work  and 
the  model,  the  organization  has  not  implemented  the  model.  Thus,  having  an  understanding 
of  the  ways  in  which  implementation  of  the  model  relates  to  what  people  are  doing  in  the 
organization  is  a  necessary  and  unavoidable  prerequisite  to  implementing  the  model.  PIlDs 
provide  a  mechanism  by  which  the  implementation  of  a  model  practice  can  be  described. 
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Application  of  Plls  in  Process  Appraisal 

During  the  course  of  process  appraisal,  the  appraisal  team’s  primary  focus  is  on  verifying 
practice  implementation.  This  verification  is  accomplished  by  (1)  obtaining  objective 
evidence  relevant  to  the  implementation  of  a  practice,  (2)  comparing  the  objective  evidence 
available  with  what  is  expected,  and  then  (3)  making  a  determination  of  practice 
implementation  based  on  the  difference  between  actual  and  expected  evidence. 

The  PIIs  assist  the  appraisal  team  (as  well  as  the  implementing  organization)  with  task  1  by 
providing  a  framework  or  structure  that  makes  explicit  the  types  of  objective  evidence  that 
should  be  considered.  In  concert  with  the  CMMI  model  documentation,  this  framework 
provides  the  model  basis  against  which  the  organization’s  actual  operating  practices  are 
compared. 

Note  that  PIIs  do  not  prescribe  what  objective  evidence  must  be  present  for  practice 
implementation  determinations  to  be  made;  they  only  make  explicit  what  is  reasonable  for  an 
appraisal  team  to  consider.  The  particular  circumstances  and  attributes  of  the  instantiation 
must  be  taken  into  consideration  when  making  determinations  of  practice  implementation.  As 
a  general  rule,  the  more  objective  evidence  and  the  more  PIIs  represented  by  that  objective 
evidence,  the  higher  the  confidence  level  that  the  practice  is  implemented. 

The  PII  structure  assists  the  appraisal  team  with  task  2  to  the  extent  that  the  team  has  agreed 
in  advance  on  the  objective  evidence  it  expects  to  see  for  each  process  instantiation 
examined.  In  some  cases  it  may  be  difficult  or  impossible  to  have  completely  developed  a 
team  consensus  on  what  objective  evidence  must  be  seen  (in  advance).  But  sooner  or  later  the 
appraisal  team  must  establish  a  consensus  view  on  what  is  reasonable  to  expect,  since  it  is 
only  the  presence  of  that  consensus  view  that  permits  a  determination  of  practice 
implementation  to  be  made. 

The  final  practice  implementation  determination  task  is  that  of  developing  a  team  consensus 
on  whether  the  practice  is  implemented  for  the  process  instantiation  being  examined.  This 
decision  is  based  on  the  difference  between  what  is  expected  and  what  is  observed. 
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Appendix  C  Alternative  Practice 

Identification  and 
Characterization  Guidance 


Concept  Description 

This  appendix  provides  guidance  on  how  to  identify  acceptable  alternatives  to  practices 
documented  in  CMMI  and  how  to  perform  practice  characterization  at  the  instantiation  and 
organizational  unit  levels  when  acceptable  alternative  practices  are  implemented  in  lieu  of 
model  practices. 

The  CMMI  Glossary  includes  the  following  definition  of  “alternative  practice. 

”A  practice  that  is  a  substitute  for  one  or  more  generic  or  specific  practices  contained  in 
CMMI  models  that  achieves  an  equivalent  effect  toward  satisfying  the  generic  or  specific 
goal  associated  with  model  practices.  Alternative  practices  are  not  necessarily  one-for-one 
replacements  for  the  generic  or  specific  practices.” 

Analysis  and  use  of  alternative  practices  in  SCAMPI  A  appraisals  involves  the  following 
activities: 

•  identifying  which  model  practices  appear  to  be  implemented  using  an  alternative 
practice,  and  analyzing  whether  or  not  the  alternative  practice  does  indeed  achieve  an 
effect  equivalent  to  that  achieved  by  the  model  practices  toward  satisfying  the  associated 
specific  or  generic  goal 

•  developing  an  instantiation-level  characterization  of  the  implementation  of  the  alternative 
practice  by  determining  whether  the  provided  evidence  includes  appropriate  direct 
artifact(s)  and  indirect  artifact(s)  or  affirmations  for  the  practice 

•  applying  the  instantiation-level  characterization  of  the  alternative  practice  to  the  model 
practice(s)  addressed 

•  aggregating  all  of  the  instantiation-level  characterizations  to  derive  the  organizational 
unit-level  characterizations  and  generating  findings  and  goal  ratings,  just  as  is  performed 
when  an  alternative  practice  has  not  been  used 
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Identification  of  Acceptable  Alternative  Practices 

Technically,  alternative  practices  can  be  discovered  at  any  time  during  an  appraisal,  up  to  and 
including  when  the  appraisal  team  is  analyzing  feedback  from  validation  of  the  preliminary 
findings.  However,  in  most  cases,  alternative  practices  are  uncovered  during  the  appraisal 
planning  and  preparation  activities.  Typically,  they  are  either  communicated  to  the  appraisal 
team  lead  by  the  organizational  unit  during  appraisal  planning  discussions  or  are  discovered 
by  the  appraisal  team  in  the  early  examinations  of  objective  evidence.  For  example,  little  or 
no  evidence  might  be  found  for  a  given  model  practice  or  set  of  related  model  practices,  but 
evidence  for  an  alternative  practice  might  be  uncovered  when  the  initial  data  set  is  analyzed 
(activity  1.4.3,  Inventory  Objective  Evidence)  or  when  a  readiness  review  is  conducted 
(activity  1.5.1,  Perform  Readiness  Review). 

The  alternative  practice  is  not  acceptable  until  the  appraisal  team  agrees  that  it  does  indeed 
achieve  an  effect  equivalent  to  one  or  more  model  practices.  To  do  so,  the  team  must  first 
analyze  the  alternative  practice  for  its  content  to  identify  which  model  practice(s)  it 
addresses.  The  appraisal  team  then  must  determine  what  effect  is  achieved  by  the 
implementation  of  the  addressed  model  practice(s)  toward  goal  satisfaction.  This 
determination  is  more  than  simply  what  work  product(s)  might  be  developed  as  a  result  of 
implementation  of  the  model  practice(s).  What  is  achieved,  supported,  and/or  enabled  as  a 
result  of  implementation  of  the  practice(s)  and  generation  of  the  work  product(s)?  What 
information  becomes  available,  when,  and  to  whom?  Once  these  questions  and  any  others  the 
team  deems  appropriate  are  answered,  the  team  would  need  to  decide  whether  the  alternative 
practice  achieves  the  same  or  similar  results  or  just-as-effective  results  for  the  given  business 
environment.  If  it  does,  then  it  achieves  an  equivalent  effect  toward  satisfying  the  goal 
associated  with  the  original  model  practice(s)  and  can  be  considered  an  acceptable 
alternative. 

As  noted  above,  an  alternative  practice  is  not  necessarily  a  one-for-one  replacement  for  a 
given  generic  or  specific  practice.  In  some  cases,  an  alternative  practice  might  only  partially 
address  one  or  more  model  practices.  In  such  cases,  the  alternative  practice  should  be 
analyzed  in  conjunction  with  the  associated  model  practice.  The  combined  implementation 
could  be  acceptable.  In  other  cases,  an  alternative  practice  might  consist  of  multiple  elements, 
all  or  some  of  which  appear  to  address  one  or  more  related  model  practices.  The  separate 
elements  would  need  to  be  examined  to  ensure  that  the  aggregate  of  those  elements  achieves 
effect(s)  toward  goal  satisfaction  equivalent  to  the  effect(s)  achieved  by  the  addressed  model 
practice(s). 


111-26 


CMU/SEI-2006-HB-002 


Instantiation-Level  Characterization  of  Alternative 
Practices 


Once  an  alternative  practice  has  been  approved  as  acceptable  by  the  appraisal  team,  its 
implementation  can  be  characterized  at  the  instantiation  level.  The  same  rules  are  used  for 
alternative  practices  as  those  applied  to  model  practices  (see  the  first  table  in  the  Parameters 
and  Limits  section  for  activity  2.2.2,  Characterize  Implementation  of  Model  Practices). 
However,  the  team  must  determine  what  artifacts  should  logically  be  expected  from 
implementation  of  the  alternative  practice.  Particular  attention  needs  to  be  paid  to  whether 
more  than  one  type  of  direct  artifact  might  be  needed  to  cover  the  complete  implementation 
approach.  This  situation  could  occur  especially  when  an  alternative  practice 

•  addresses  more  than  one  model  practice 

•  provides  only  partial  coverage  of  a  model  practice  and  is  combined  with  the  remaining 
aspects  of  that  model  practice  to  create  an  acceptable  alternative  practice 

•  consists  of  multiple  elements 

After  the  appraisal  team  decides  what  artifacts  it  should  expect  to  see,  the  team  can  examine 
the  evidence  provided  to  determine  its  adequacy  and  coverage  of  the  alternative  practice  to 
characterize  the  implementation  of  that  practice. 

Instantiation-Level  Characterization  of  the  Associated 
Model  Practice(s) 

Typically,  the  characterization  of  the  alternative  practice  is  applied  directly  to  the  addressed 
model  practices.  This  characterization  is  straightforward  in  cases  where  the  alternative 
practice  addresses  a  single  model  practice.  However,  in  cases  where  an  alternative  practice 
addresses  more  than  one  model  practice,  the  characterizations  of  the  model  practices  may 
vary  depending  on  the  weaknesses  documented  for  the  alternative  practice  and  whether  the 
evidence  provided  fully  covers  the  alternative  practice  or  not.  The  weaknesses  associated 
with  the  alternative  practice  might  apply  to  only  some  of  the  addressed  model  practices.  For 
example,  in  a  case  where  an  alternative  practice  is  characterized  as  Largely  Implemented  or 
Partially  Implemented,  some  of  the  addressed  model  practices  could  still  be  characterized 
higher  if  none  of  the  weaknesses  apply  to  those  model  practice(s)  and  the  provided  evidence 
is  adequate  for  those  practices. 

Organizational  Unit-Level  Characterization  and  Goal 
Rating 

Once  the  addressed  model  practices  have  been  characterized  at  the  instantiation  level, 
aggregation  of  the  instantiation- level  practice  characterizations  proceeds  just  as  is 
documented  in  the  second  table  in  the  Parameters  and  Limits  for  activity  2.4.2,  Characterize 
Implementation  of  Model  Practices.  In  addition,  goal  rating  is  performed  just  as  is 
documented  in  activity  2.6.1,  Derive  Findings  and  Rate  Goals. 
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Appendix  D  ARC/MDD  Traceability  Table 


Table  111-4:  ARC  Traceability 


ARC  ID 

ARC  Requirement 

MDD  Process 

MDD  Activity 

Notes/Comments 

4 

Requirements  for  CMMI  Appraisal  Methods 

4.1 

Responsibilities 

Key  roles  and  responsibilities  are  addressed 
in  process  1 .3,  Select  and  Prepare  Team. 

4.1.1 

The  method  shall  define  the  responsibilities  of  the  appraisal 
sponsor,  which  at  a  minimum  shall  include  the  following 
activities: 

4.1. l.a 

(  ABC)  Verify  that  the  appraisal  team  leader  has  the  appropriate 
experience,  knowledge,  and  skills  to  take  responsibility  for  and 
lead  the  appraisal. 

1 .3  Select  and  Prepare 
Team 

1.3.1  Identify  Appraisal  Team 
Leader 

4.1. l.b 

(ABC)  Ensure  that  the  appropriate  organizational  units  or 
subunits  (e.g.,  projects,  functional  units)  participate  in  the 
appraisal. 

1 . 1  Analyze  Requirements 

1.1.3  Detennine  Appraisal 
Scope 

4.1. l.c 

(ABC)  Support  appraisal  method  provisions  for  ensuring  non¬ 
attribution  to  appraisal  participants. 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

Non-attribution  is  also  addressed  throughout 
the  MDD  in  discussions  of  team  members 
(1.3.3),  preliminary  findings  (2.5.1), 
documentation  of  objective  evidence  (2.3), 
reporting  (3.1),  and  recording  (3.2). 

4.1. l.d 

(ABC)  Ensure  that  resources  are  made  available  to  conduct  the 
appraisal. 

1.2  Develop  Appraisal 

Plan 

1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

Resources  are  identified  in  several  sections 
of  1.2,  Develop  Appraisal  Plan,  and 
commitment  is  obtained  in  1 .2.6. 

4.1.  l.e 

(ABC)  Review  and  approve  the  appraisal  input  prior  to  the 
beginning  of  data  collection  by  the  appraisal  team. 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 
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ARC  ID 

ARC  Requirement 

MDD  Process 

MDD  Activity 

Notes/Comments 

4.1.2 

The  method  shall  define  the  responsibilities  of  the  appraisal 
team  leader,  which  at  a  minimum  shall  include  the  following 
activities: 

1 .3  Select  and  Prepare 
Team 

1.3.1  Identify  Appraisal  Team 
Leader 

Appraisal  team  leader  responsibilities  are 
defined  throughout,  but  are  summarized  in 
1.3.1. 

4. 1.2. a 

(  ABC)  Ensure  that  the  appraisal  is  conducted  in  accordance 
with  the  method’s  documented  process. 

1.2  Develop  Appraisal 

Plan 

1.2.1  Tailor  Method 

4.1.2.b 

(ABC)  Confirm  the  sponsor’s  commitment  to  proceed  with  the 
appraisal. 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

The  primary  mechanism  to  describe  and 
document  sponsor  commitment  and  appraisal 
objectives  is  the  appraisal  input  (1.1.5). 

4.1.2.C 

(ABC)  Ensure  that  appraisal  participants  are  briefed  on  the 
purpose,  scope,  and  approach  of  the  appraisal. 

2.1  Prepare  Participants 

2.1.1  Conduct  Participant 
Briefing 

4.1.2.d 

(ABC)  Ensure  that  all  appraisal  team  members  have  the 
appropriate  experience,  knowledge,  and  skills  in  the  appraisal 
reference  model  and  appraisal  method;  the  necessary 
competence  to  use  instruments  or  tools  chosen  to  support  the 
appraisal;  and  access  to  documented  guidance  on  how  to 
perfonn  the  defined  appraisal  activities. 

1 .3  Select  and  Prepare 
Team 

1.3.3  Prepare  Team 

Also  addressed  by  selection  of  team 
members  with  appropriate  qualifications  in 
1.3.2. 

4.1. 2.e 

( ABC)Verify  and  document  that  the  appraisal  method 
requirements  have  been  met. 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

4.2 

Appraisal  Method  Documentation 

4.2.1 

The  method  shall  be  documented  and,  at  a  minimum,  include 

MDD,  VI. 2 

All 

4.2.1. a 

(ABC)  identification  of  the  CMMI  models  (version,  discipline, 
and  representation  [staged  or  continuous])  with  which  the 
method  can  be  used 

4.2. l.b 

(ABC)  identification  of  the  ARC  version  on  which  the 
appraisal  method  is  based 

Method  Context 

ARC  VI. 2 

4.2.1.C 

(ABC)  identification  of  which  CMMI  appraisal  requirements 
are  satisfied  by  the  method,  along  with  the  CMMI  appraisal 
class  membership  (if  applicable) 

Method  Context 

SCAMPI  addresses  all  ARC  Class  A  method 
requirements. 

4.2. l.d 

(ABC)  activity  descriptions,  artifacts,  and  guidance  that 
implement  each  of  the  appraisal  requirements 

(All  phases,  processes, 
activities) 

MDD  process  descriptions  in  Part  II. 

4.2. l.e 

(A)  declaration  as  to  whether  or  not  the  method  supports 
15504-conformant  appraisals 

Executive  Summary 
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ARC  ID 

ARC  Requirement 

MDD  Process 

MDD  Activity 

Notes/Comments 

4.2.2 

The  method  documentation  shall  provide  guidance  for 

4. 2. 2. a 

(ABC)  identifying  an  appraisal's  purpose,  objectives,  and 
constraints 

1 . 1  Analyze  Requirements 

1.1.1  Detennine  Appraisal 
Objectives 

4.2. 2. b 

(ABC)  determining  the  suitability  of  the  appraisal  method 
relative  to  the  appraisal’s  purpose,  objectives,  and  constraints 

1 . 1  Analyze  Requirements 

1.1.1  Detennine  Appraisal 
Objectives 

Also  addressed  by  commitment  to  appraisal 
input  (1.1.5)  and  selection  of  appraisal  usage 
mode  (Modes  of  Usage). 

4.2.3 

The  method  documentation  shall  provide  guidance  for 
identifying  the  scope  of  the  CMMI  model(s)  to  be  used  for  the 
appraisal: 

1 . 1  Analyze  Requirements 

1.1.3  Detennine  Appraisal 
Scope 

4.2.3. a 

(ABC)  process  areas  to  be  investigated  (continuous  and  staged 
representations) 

1 . 1  Analyze  Requirements 

1.1.3  Detennine  Appraisal 
Scope 

4.2. 3. b 

(ABC)  capability  levels  to  be  investigated  for  each  process 
area  (continuous  representation) 

1 . 1  Analyze  Requirements 

1.1.3  Detennine  Appraisal 
Scope 

4.2.4 

The  method  documentation  shall  provide  guidance  for 
identifying  the  organizational  unit  to  be  appraised: 

1 . 1  Analyze  Requirements 

1.1.3  Detennine  Appraisal 
Scope 

4.2.4.a 

(ABC)  the  sponsor  of  the  appraisal  and  the  sponsor’s 
relationship  to  the  organizational  unit  being  appraised 

1 . 1  Analyze  Requirements 

1.1.1  Detennine  Appraisal 
Goals 

4.2. 4. b 

(ABC)  projects  within  the  organizational  unit  that  will 
participate 

1 . 1  Analyze  Requirements 

1.1.3  Detennine  Appraisal 
Scope 

Selection  of  sample  projects  and  support 
groups  that  are  representative  of  the 
organizational  unit  is  addressed  by  the 
appraisal  input  (1.1)  and  appraisal  plan  (1.2). 

4.2.4.C 

(ABC)  functional  elements  of  the  organizational  unit  that  will 
participate 

1 . 1  Analyze  Requirements 

1.1.3  Detennine  Appraisal 
Scope 

Selection  of  sample  projects  and  support 
groups  that  are  representative  of  the 
organizational  unit  is  addressed  by  the 
appraisal  input  (1.1)  and  appraisal  plan  (1.2). 

4.2. 4.d 

(ABC)  names  and  affiliations  (organizational  units)  of 
participants  in  the  appraisal  activities 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

Appraisal  participants  are  among  the 
resources  identified  in  1.2.2. 

4.2.5 

The  method  documentation  shall  provide  guidance  for 
selecting  appraisal  team  members  and  criteria  for  qualification 
including 

1 .3  Select  and  Prepare 
Team 

1.3.2  Select  Team  Members 

4. 2. 5. a 

(ABC)  technical  experience  (discipline-specific) 

1 .3  Select  and  Prepare 
Team 

1.3.2  Select  Team  Members 
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ARC  ID 

ARC  Requirement 

MDD  Process 

MDD  Activity 

Notes/Comments 

4.2. 5.b 

(ABC)  management  experience 

1 .3  Select  and  Prepare 
Team 

1.3.2  Select  Team  Members 

4.2.5.C 

(ABC)  experience,  knowledge,  and  skills  in  the  appraisal 
reference  model  and  appraisal  method 

1 .3  Select  and  Prepare 
Team 

1.3.2  Select  Team  Members 

4.2.6 

The  method  documentation  shall  provide  guidance  for  an 
appraisal  team  leader’s  qualification  criteria,  including 

1 .3  Select  and  Prepare 
Team 

1.3.1  Identify  Appraisal  Team 
Leader 

Requirements  for  SCAMPI  Lead  Appraisers 
are  defined  and  verified  via  the  SEI 

Appraisal  Program.  Specific  qualifications 
and  requirements  are  available  on  the  SEI 
Web  site. 

4. 2. 6. a 

(ABC)  training  and  experience  using  the  appraisal  reference 
model 

1 .3  Select  and  Prepare 
Team 

1.3.1  Identify  Appraisal  Team 
Leader 

4.2. 6. b 

(ABC)  training  and  experience  using  the  appraisal  method 

1 .3  Select  and  Prepare 
Team 

1.3.1  Identify  Appraisal  Team 
Leader 

4.2.6.C 

(ABC)  experience  in  delivering  training,  managing  teams, 
facilitating  group  discussions,  and  making  presentations 

1 .3  Select  and  Prepare 
Team 

1.3.1  Identify  Appraisal  Team 
Leader 

This  requirement  is  also  addressed  by  the 
SCAMPI  Lead  Appraiser  candidate  selection 
criteria  published  on  the  SEI  Web. 

4.2.7 

(ABC)  The  method  documentation  shall  provide  guidance  for 
determining  the  appropriate  size  of  the  appraisal  team.  For 
Class  A  appraisals,  the  minimum  team  size  is  four  members; 
for  Class  B  appraisals,  two  members;  for  Class  C  appraisals, 
one  member. 

1 .3  Select  and  Prepare 
Team 

1.3.2  Select  Team  Members 

4.2.8 

(ABC)  The  method  documentation  shall  provide  guidance  on 
the  roles  and  responsibilities  of  appraisal  team  members. 

1 .3  Select  and  Prepare 
Team 

1.3.2  Select  Team  Members 

4.2.9 

(ABC)  The  method  documentation  shall  provide  guidance 
addressing  the  responsibilities  of  the  appraisal  sponsor. 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

Sponsor  responsibilities  are  throughout,  but 
are  primarily  defined  in  1.1,  Analyze 
Requirements,  and  1 .2,  Develop  Appraisal 
Plan. 

4.2.10 

(ABC)  The  method  documentation  shall  provide  guidance 
addressing  the  responsibilities  of  the  appraisal  team  leader. 

1 .3  Select  and  Prepare 
Team 

1.3.1  Identify  Appraisal  Team 
Leader 

Appraisal  team  leader  responsibilities  are 
defined  throughout,  but  are  summarized  in 
1.3.1. 
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ARC  ID 

ARC  Requirement 

MDD  Process 

MDD  Activity 

Notes/Comments 

4.2.11 

(ABC)  The  method  documentation  shall  provide  guidance  for 
estimating  the  resources  required  to  conduct  the  appraisal 
(including  the  amount  of  time  required  to  conduct  an 
appraisal). 

1.2  Develop  Appraisal 

Plan 

1.2.3  Detennine  Cost  and 
Schedule 

Estimates  of  appraisal  resources  are 
addressed  throughout  development  of  the 
appraisal  plan  in  1.2. 

4.2.12 

(ABC)  The  method  documentation  shall  provide  guidance  for 
appraisal  logistics. 

1.2  Develop  Appraisal 

Plan 

1 .2.4  Plan  and  Manage 
Logistics 

4.2.13 

(ABC)  The  method  documentation  shall  provide  guidance  for 
collecting  relevant  data  on  the  organizational  unit  and 
associating  the  data  to  the  specific  and  generic  practices  of  the 
appraisal  reference  model. 

2.2  Examine  Objective 
Evidence 

2.2 

Addressed  by  individual  sections  of  2.2 
related  to  sources  of  objective  evidence. 

4.2.14 

(ABC)  The  method  documentation  shall  provide  guidance  for 
creating  findings,  including  both  strengths  and  weaknesses 
relative  to  the  appraisal  reference  model. 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

Addressed  in  descriptions  of  preliminary 
findings  (2.5.1),  final  findings  (2.6.4),  and 
crafting  strengths  and  weaknesses  (2.4.1).  In 
SCAMPI  VI. 2,  the  emphasis  is  on 
identifying  weaknesses  and  significant 
strengths  that  are  expected  to  become  part  of 
the  findings.  Recording  of  satisfactory 
implementations  is  done  by  verifying  PIIs 
(2.4)  rather  than  by  text  statements. 

4.2.15 

(ABC)  The  method  documentation  shall  provide  guidance  for 
protecting  the  confidentiality  of  appraisal  data  and  ensuring 
non-attribution  of  data  contributed  by  appraisal  participants. 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.4  Archive  and/or  Dispose 
of  Key  Artifacts 

Confidentiality  and  non-attribution  principles 
are  addressed  throughout  the  MDD  in 
discussions  of  team  members  (1.3.3), 
preliminary  findings  (2.5.1),  documentation 
of  objective  evidence  (2.3),  reporting  (3.1), 
and  recording  (3.2). 

4.2.16 

The  method  documentation  shall  provide  guidance:  for  ( 1 ) 
recording  traceability  between  the  data  collected  during  the 
appraisal  and  the  findings  and/or  ratings,  (2)  the  retention  and 
safekeeping  of  appraisal  records,  and  (3)  compiling  and 
maintaining  an  appraisal  record  that  supports  the  appraisal 
team’s  findings  and/or  ratings  and  that  contains  the  following 
minimum  content: 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

See  section  3.2.2  for  the  description  and 
contents  of  the  appraisal  record. 

4.2.16.a 

(ABC)  dates  of  appraisal 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 
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4.2.16.b 

(ABC)  appraisal  input 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

The  appraisal  record  includes  the  latest 
version  of  the  appraisal  input,  which  was 
originally  agreed  to  by  the  sponsor  in  1.1.5. 

4.2.16.C 

(A)  objective  evidence,  or  identification  thereof,  sufficient  to 
substantiate  goal  rating  judgments 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

As  described  in  3.2.2,  this  activity  may  be  an 
identification  of  the  objective  evidence  rather 
than  a  full  or  partial  copy  of  the  actual 
evidence.  A  suitable  implementation  for  the 
intent  of  this  requirement  might  be  the  set  of 
PIIs  used  for  practice  characterization  in 
2.4.2. 

4.2.16.d 

(ABC)  identification  of  appraisal  method  (and  version)  used, 
along  with  any  tailoring  options 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

See  also  the  Appraisal  Disclosure  Statement 
(ADS)  described  in  Appendix  A. 

4.2.16.e 

(ABC)  findings 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

4.2.16.f 

(A)  any  ratings  rendered  during  the  appraisal  (goals,  process 
areas,  and  maturity  or  capability  levels) 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 

4.3 

Planning  and  Preparing  for  the  Appraisal 

4.3.1 

The  method  shall  provide  for  the  preparation  of  appraisal 
participants  by  addressing,  at  a  minimum, 

2.1  Prepare  Participants 

2.1.1  Conduct  Participant 
Briefing 

4.3. l.a 

(ABC)  the  purpose  of  the  appraisal 

2.1  Prepare  Participants 

2.1.1  Conduct  Participant 
Briefing 

4.3. l.b 

(ABC)  the  scope  of  the  appraisal 

2.1  Prepare  Participants 

2.1.1  Conduct  Participant 
Briefing 

4.3. l.c 

(ABC)  the  appraisal  approach 

2.1  Prepare  Participants 

2.1.1  Conduct  Participant 
Briefing 

4.3. l.d 

(ABC)  the  roles  and  responsibilities  of  participants  in  the 
appraisal 

2.1  Prepare  Participants 

2.1.1  Conduct  Participant 
Briefing 

4.3. l.e 

(ABC)  the  schedule  of  appraisal  activities 

2.1  Prepare  Participants 

2.1.1  Conduct  Participant 
Briefing 

4.3.2 

(ABC)  The  method  shall  provide  for  the  development  of  the 
appraisal  input  prior  to  the  beginning  of  data  collection  by  the 
appraisal  team. 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

The  appraisal  input  may  be  generated 
incrementally  throughout  planning,  but  must 
be  approved  prior  to  the  start  of  data 
collection. 

111-34 


CMU/SEI-2006-HB-002 


ARC  ID 

ARC  Requirement 

MDD  Process 

MDD  Activity 

Notes/Comments 

4.3.3 

At  a  minimum,  the  appraisal  input  shall  specify 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

Contents  of  the  appraisal  input  are  described 
throughout  section  1.1.  The  appraisal  input  is 
approved  by  the  sponsor  in  1.1.5. 

4.3. 3. a 

(ABC)  the  identity  of  the  sponsor  of  the  appraisal,  and  the 
sponsor’s  relationship  to  the  organizational  unit  being 
appraised 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3. 3. b 

(  ABC)  the  appraisal  purpose,  including  alignment  with 
business  objectives 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

See  also  1.1.1,  Determine  Appraisal 
Objectives. 

4.3.3.C 

(ABC)  the  appraisal  reference  model  scope,  including 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

See  also  1.1.3,  Detennine  Appraisal  Scope. 

4.3.3.C.1 

the  process  areas  to  be  investigated  within  the  organizational 
unit 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3.3.C.2 

the  highest  maturity  level  and/or  capability  level  to  be 
investigated  for  each  process  area  within  the  appraisal  scope 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3. 3. d 

(ABC)  the  organizational  unit  that  is  the  subject  of  the 
appraisal 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

See  also  1.1.3,  Detennine  Appraisal  Scope. 

4.3.3.e 

(ABC)  the  process  context,  which,  at  a  minimum,  shall  include 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3.3.e.l 

the  size  of  the  organizational  unit 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3.3.e.2 

the  demographics  of  the  organizational  unit 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3.3.e.3 

the  application  domain  of  the  products  or  services  of  the 
organizational  unit 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3.3.e.4 

the  size,  criticality,  and  complexity  of  the  products  or  services 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3. 3. f 

(  ABC)  the  appraisal  constraints,  which,  at  a  minimum,  shall 
include 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

See  also  1.1.2,  Detennine  Appraisal 
Constraints. 

4.3.3.f.  1 

availability  of  key  resources  (e.g.,  staffing,  funding,  tools, 
facilities) 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3.3.f.2 

schedule  constraints 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 
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4.3.3.f.3 

the  maximum  amount  of  time  to  be  used  for  the  appraisal 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3. 3. f.  4 

specific  process  areas  or  organizational  entities  to  be  excluded 
from  the  appraisal 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3.3.f.5 

the  minimum,  maximum,  or  specific  sample  size  or  coverage 
that  is  desired  for  the  appraisal 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3.3.f.6 

the  ownership  of  the  appraisal  outputs  and  any  restrictions  on 
their  use 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3.3.f.7 

controls  on  information  resulting  from  a  confidentiality 
agreement 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3.3.f.8 

non-attribution  of  appraisal  data  to  associated  sources 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3. 3. g 

(ABC)  the  identity  of  the  CMMI  models  used,  including  the 
version,  discipline,  and  representation  (staged  or  continuous) 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

See  also  1.1.3,  Determine  Appraisal  Scope. 

4.3. 3.h 

(ABC)  the  criteria  for  experience,  knowledge,  and  skills  of  the 
appraisal  team  leader  who  is  responsible  for  the  appraisal 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

See  also  1.3.1,  Identify  Appraisal  Team 
Leader. 

4.3. 3. i 

(ABC)  the  identity  and  affiliation  of  the  appraisal  team 
members,  including  the  appraisal  team  leader,  with  their 
specific  responsibilities  for  the  appraisal 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3.3.j 

(ABC)  the  identity  (name  and  organizational  affiliation)  of 
appraisal  participants  and  support  staff,  with  specific 
responsibilities  for  the  appraisal 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

See  also  1 .2.2,  Identify  Needed  Resources 

4.3. 3.k 

(ABC)  any  additional  information  to  be  collected  during  the 
appraisal  to  support  achievement  of  the  appraisal  objectives 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

See  also  1.1.4,  Determine  Outputs. 

4.3.3.1 

(ABC)  a  description  of  the  planned  appraisal  outputs,  including 
ratings  to  be  generated  (process  areas,  maturity  level) 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

See  also  1.1.4,  Detennine  Outputs. 

4.3. 3. m 

(ABC)  anticipated  follow-on  activities  (e.g.,  reports,  appraisal 
action  plans,  re-appraisal) 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

See  also  1.1.4,  Detennine  Outputs. 

4.3.3.n 

(ABC)  planned  tailoring  of  the  appraisal  method  and 
associated  tradeoffs,  including  the  sample  size  or  coverage  of 
the  organizational  unit 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

See  also  1.2.1,  Tailor  Method. 
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4.3.4 

(ABC)  The  method  shall  require  that  the  appraisal  input,  and 
any  changes  to  the  appraisal  input,  shall  be  agreed  to  by  the 
sponsor  (or  the  delegated  authority)  and  documented  in  the 
appraisal  record. 

1 . 1  Analyze  Requirements 

1.1.5  Obtain  Commitment  to 
Appraisal  Input 

4.3.5 

The  method  shall  require  the  development  of  an  appraisal  plan 
that,  at  a  minimum,  specifies 

1.2  Develop  Appraisal 

Plan 

1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

The  appraisal  plan  is  described  throughout 
Section  1.2.  Completion  and  agreement  of 
plan  contents  is  described  in  1.2.6. 

4.3. 5. a 

(ABC)  the  appraisal  input 

1.2  Develop  Appraisal 

Plan 

1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

4.3. 5. b 

(ABC)  the  activities  to  be  performed  in  conducting  the 
appraisal 

1.2  Develop  Appraisal 

Plan 

1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

4.3.5.C 

(ABC)  resources  and  schedule  assigned  to  appraisal  activities 

1.2  Develop  Appraisal 

Plan 

1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

See  also  1 .2.2,  Identify  Needed  Resources, 
and  1.2.3,  Determine  Cost  and  Schedule. 

4.3. 5. d 

(ABC)  appraisal  logistics 

1.2  Develop  Appraisal 

Plan 

1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

See  also  1 .2.4,  Plan  and  Manage  Logistics. 

4.3.5.e 

(ABC)  mitigation  steps  to  address  risks  associated  with 
appraisal  execution 

1.2  Develop  Appraisal 

Plan 

1.2.6  Obtain  Commitment  to 
Appraisal  Plan 

See  also  1.2.5,  Document  and  Manage  Risks. 

4.4 

Appraisal  Data  Collection 

4.4. intro 

Appraisal  teams  base  their  findings  on  review  of  one  or  more 
types  of  objective  evidence.  The  requirements  in  this  section 
identify  the  types  of  objective  evidence  recognized  by  CMMI 
appraisal  methods.  As  indicated  in  Appendix  A,  two  types  of 
objective  evidence  identified  below  are  required  for  Class  A 
and  Class  B  appraisal  methods.  At  least  one  type  of  objective 
evidence  is  required  for  Class  C  methods. 

4.4.1 

(See  Appendix  A)  The  method  shall  collect  data  by  conducting 
interviews  (e.g.,  with  project  leaders,  managers,  practitioners). 

2.2  Examine  Objective 
Evidence 

2.2.2  Examine  Objective 
Evidence  from  Interviews 

4.4.2 

(See  Appendix  A)  The  method  shall  collect  data  by  reviewing 
documentation  (e.g.,  organizational  policies,  instruments, 
project  procedures,  and  implementation-level  work  products). 

2.2  Examine  Objective 
Evidence 

2.2.1  Examine  Objective 
Evidence  from  Documents 

4.5 

Data  Consolidation  and  Validation 
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4.5.1 

(ABC)  The  method  shall  require  appraisal  team  consensus 
when  teams  are  involved  in  decisions  determining  the  validity 
of  findings  and  establishing  ratings. 

2.6  Generate  Appraisal 
Results 

2.6.1  Derive  Findings  and 

Rate  Goals 

Use  of  consensus  as  a  team  decision-making 
technique  is  discussed  throughout  applicable 
sections  of  the  MDD.  A  summary  of 
consensus  decisions  needed  is  depicted  in 
"Data  Collection,  Rating,  and  Reporting.” 

4.5.2 

The  method  shall  require  a  mechanism  for  consolidating  the 
data  collected  during  an  appraisal  into  accurate  findings 
according  to  the  following  criteria: 

2.3  Document  Objective 
Evidence 

2.3.3  Document  Practice 
Implementation 

4. 5. 2. a 

(ABC)  The  finding  was  derived  from  objective  evidence  seen 
or  heard  during  data  collection  sessions. 

2.3  Document  Objective 
Evidence 

2.3.3  Document  Practice 
Implementation 

See  also  descriptions  of  verifying  practice 
implementation  indicator  types  (direct, 
indirect,  affirmation)  in  2.4. 1 . 

4.5. 2.b 

(ABC)  The  finding  is  clearly  worded,  phrased  without 
attribution,  and  expressed  in  terminology  used  at  the 
organizational  unit. 

2.3  Document  Objective 
Evidence 

2.3.3  Document  Practice 
Implementation 

4.5.2.C 

(ABC)  Objective  evidence  supporting  the  finding  is  traceable 
to  the  project  or  organizational  unit. 

2.3  Document  Objective 
Evidence 

2.3.3  Document  Practice 
Implementation 

4.5. 2.d 

(  ABC)  The  finding  is  relevant  to  the  appraisal  reference  model 
and  can  be  associated  with  a  specific  model  component. 

2.3  Document  Objective 
Evidence 

2.3.3  Document  Practice 
Implementation 

4.5.3 

The  method  shall  require  a  mechanism  for  validating  findings 
according  to  the  following  criteria: 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

All  sections  of  2.4  apply. 

4. 5. 3. a 

( AB)  The  finding  is  based  on  corroborated  objective  evidence. 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

In  SCAMPI  VI. 2,  corroboration  is  addressed 
by  method  requirements  for  a  combination  of 
indicator  types  (direct,  indirect,  affirmation) 
as  described  in  2.4.1. 

4.5.3.b 

( AB)  The  finding  is  consistent  with  other  verified  findings. 
(Verified  findings  cannot  be  both  true  and  mutually 
inconsistent;  in  aggregate,  they  constitute  a  set  of  truths  about 
the  organizational  unit  that  must  be  consistent.) 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

In  SCAMPI  VI. 2,  consistency  is  addressed 
by  method  requirements  for  a  combination  of 
indicator  types  (direct,  indirect,  affirmation) 
as  described  in  2.4.1. 

4.5.4 

The  method  shall  require  the  following  minimum  set  of  criteria 
to  be  satisfied  in  order  for  objective  evidence  to  be  considered 
“corroborated”: 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 
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4.5.4.a 

(AB)  The  objective  evidence  is  obtained  from  at  least  two 
different  sources. 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

In  SCAMPI  VI. 2,  corroboration  is  addressed 
by  method  requirements  for  a  combination  of 
indicator  types  (direct,  indirect,  affirmation) 
as  described  in  2.4.1. 

4.5.4.b 

( AB)  At  least  one  of  the  two  sources  must  reflect  work  actually 
being  done  (e.g.,  process  area  implementation). 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

In  SCAMPI  VI. 2,  this  activity  is  addressed 
by  requirements  for  a  direct  artifact  from 
each  instantiation  for  each  practice  (2.4.1). 
Affirmations  also  reflect  work  being  done, 
but  are  not  necessarily  required  from  each 
instantiation  for  each  practice;  affirmations 
are  used  to  corroborate  direct  artifacts. 

4.5.5 

The  method  shall  require  a  mechanism  for  determining  that 
sufficient  data  has  been  collected  to  cover  the  scope  of  the 
appraisal,  according  to  the  following  minimum  set  of  rules: 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

Coverage  is  addressed  by  requirements  for 
objective  evidence  from  each  instantiation 
for  each  practice  (2.4.1).  See  also  1.5.3, 
Replan  Data  Collection,  for  collection  of 
additional  objective  evidence  necessary  to 
obtain  sufficient  coverage. 

4.5. 5. a 

(A)  A  specific  or  generic  practice  has  sufficient  data  coverage 
if  corroborated  objective  evidence  exists  for  the  practice  and 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

4.5.5.a.l 

is  adequate  to  understand  the  extent  of  implementation  of  the 
practice 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

4.5.5.a,2 

is  representative  of  the  organizational  unit 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

4.5.5.a.3 

is  representative  of  the  lifecycle  phases  in  use  within  the 
organizational  unit 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

See  1.1.3  for  selection  of  projects  and 
support  groups  representative  of  the 
organizational  unit. 

4.5. 5.b 

(A)  In  a  staged  representation,  a  process  area  has  sufficient 
data  coverage  if  all  of  its  specific  and  generic  practices  have 
sufficient  data  coverage. 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

See  also  descriptions  of  documenting 
objective  evidence  in  2.3,  and  rating  in  2.6. 
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4.5.5.C 

(A)  In  a  continuous  representation,  a  process  area  has 
sufficient  data  coverage  if  all  of  its  specific  practices  and  the 
generic  practices  within  the  appraisal  scope  have  sufficient 
data  coverage  up  through  the  capability  level  being 
investigated  for  the  process  area  (e.g.,  the  target  capability 
level). 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

4.5.6 

(A)  The  method  shall  require  a  mechanism  for  consolidating 
objective  evidence  into  preliminary  findings  of  strengths  and 
weaknesses  relative  to  the  appraisal  reference  model. 

2.4  Verify  Objective 
Evidence 

2.4.1  Verify  Objective 
Evidence 

Preliminary  findings  are  described  in  2.4.1. 

4.5.7 

(A)  The  method  shall  require  that  the  appraisal  participants  be 
presented  with  the  preliminary  findings  to  solicit  their 
responses  for  validation  of  the  findings’  accuracy  and  clarity. 

2.5  Validate  Preliminary 
Findings 

2.5.1  Validate  Practice 
Implementation  Gaps 

Validation  of  preliminary  findings  is 
addressed  in  2.5.1. 

4.6 

Rating 

4.6.1 

The  method  shall  define  a  rating  process  that  specifies,  at  a 
minimum,  the  following: 

2.6  Generate  Appraisal 
Results 

2.6* 

Descriptions  of  rating  are  addressed  by  all 
activities  in  section  2.6.  Variants  for  rating 
using  staged  and  continuous  representations 
are  provided. 

4.6.1. a 

(A)  An  appraisal  team  can  rate  a  specific  or  generic  goal  when 
corroborated  objective  evidence  for  each  practice  related  to  the 
goal  meets  the  method’s  defined  data  coverage  criteria. 

2.6  Generate  Appraisal 
Results 

2.6.1  Derive  Findings  and 

Rate  Goals 

See  also  descriptions  relating  to  collection 
and  verification  of  objective  evidence  (2.4) 
and  sufficiency  of  coverage  (2.3). 

4.6.  l.b 

(A)  An  appraisal  team  can  rate  a  process  area  when  it  has  rated 
each  of  the  process  area’s  specific  goals  and  generic  goals 
within  the  appraisal  scope. 

2.6  Generate  Appraisal 
Results 

2.6.2a  Determine  Process 

Area  Capability  Level 

2.6.2b  Determine  Satisfaction 
of  Process  Areas 

4.6.1.C 

(A)An  appraisal  team  can  determine  a  maturity  level  rating 
once  it  has  rated  all  of  the  process  areas  within  that  level  and 
each  level  below. 

2.6  Generate  Appraisal 
Results 

2.6.3b  Determine  Maturity 
Level 

4.6. l.d 

(A)  An  appraisal  team  can  determine  the  capability  level  of  a 
process  area  when  it  has  rated  each  of  the  generic  goals  at  or 
below  the  target  capability  level. 

2.6  Generate  Appraisal 
Results 

2.6.2a  Determine  Process 

Area  Capability  Level 
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ARC  ID 

ARC  Requirement 

MDD  Process 

MDD  Activity 

Notes/Comments 

4.6.2 

(A)The  method  shall  require  that  maturity  level  ratings  and/or 
capability  level  ratings  be  based  on  the  definitions  of  capability 
levels  and  maturity  levels  in  the  CMMI  models. 

2.6  Generate  Appraisal 
Results 

2.6.3a  Determine  Capability 
Profile 

2.6.3b  Determine  Maturity 
Level 

Also  see  “Data  Collection,  Rating,  and 
Reporting.” 

4.6.3 

The  method  shall  rate  each  specific  and  generic  goal  (provided 
the  prerequisites  of  rating  have  been  completed)  within  the 
appraisal  scope  in  accordance  with  the  following  rules: 

2.6  Generate  Appraisal 
Results 

2.6.1  Derive  Findings  and 

Rate  Goals 

4.6.3. a 

(A)  Rate  the  goal  “satisfied”  when  the  associated  generic  or 
specific  practices  (or  acceptable  alternative  practices)  are 
judged  to  be  implemented  and  the  aggregate  of  weaknesses 
does  not  have  a  significant  negative  impact  on  goal 
achievement. 

2.6  Generate  Appraisal 
Results 

2.6.1  Derive  Findings  and 

Rate  Goals 

4.6. 3. b 

(A)  Rate  the  goal  “not  rated”  if  the  goals  cannot  be  rated  in 
accordance  with  the  method’s  defined  criteria  for  data 
sufficiency. 

2.6  Generate  Appraisal 
Results 

2.6.1  Derive  Findings  and 

Rate  Goals 

4.6.4 

The  method  shall  rate  each  process  area  within  the  appraisal 
scope,  if  requested  by  the  appraisal  sponsor,  in  accordance 
with  the  following  rules: 

2.6  Generate  Appraisal 
Results 

2.6.2a  Determine  Process 

Area  Capability  Level 

2.6.2b  Determine  Satisfaction 
of  Process  Areas 

4.6.4.a 

(A)  For  a  staged  representation,  the  process  area  is  “satisfied” 
if  and  only  if  all  of  its  specific  and  generic  goals  are  rated 
“satisfied.” 

2.6  Generate  Appraisal 
Results 

2.6.2b  Determine  Satisfaction 
of  Process  Areas 

4.6. 4.b 

(A)  When  an  applicable  process  area  is  outside  the  scope  of  the 
model  used  for  the  appraisal,  the  process  area  is  designated  as 
“out  of  scope”  and  is  not  rated. 

2.6  Generate  Appraisal 
Results 

2.6.2b  Determine  Process 
Areas 

4.6.4.C 

(A)  When  one  or  more  goals  cannot  be  outside  of  the 
organizational  unit’s  scope  of  work,  the  process  area  is 
designated  as  “not  rated”  and  is  not  rated. 

2.6  Generate  Appraisal 
Results 

2.6.2a  Determine  Process 

Area  Capability  Level 

2.6.2b  Determine  Satisfaction 
of  Process  Areas 

4.6. 4.d 

(A)  Otherwise,  when  a  process  area  is  to  be  rated  for  a  staged 
representation,  the  process  area  is  “satisfied”  if  and  only  if  all 
of  its  specific  and  generic  goals  at  the  maturity  level  of  interest 
and  below  are  rated  “satisfied.”  Else,  it  is  “unsatisfied.”. 

2.6  Generate  Appraisal 
Results 

2.6.2b  Determine  Satisfaction 
of  Process  Areas 
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ARC  ID 

ARC  Requirement 

MDD  Process 

MDD  Activity 

Notes/Comments 

4.6.4.e 

(A)  Otherwise,  when  a  process  area  is  to  be  rated  for  a 
continuous  representation,  the  process  area  is  given  a 
capability  level  rating  based  on  the  highest  level  for  which  its 
specific  goals  and  the  generic  goals  have  been  satisfied. 

2.6  Generate  Appraisal 
Results 

2.6.2a  Determine  Process 

Area  Capability  Level 

4.6.5 

The  method  shall  rate  the  maturity  level,  if  requested  by  the 
appraisal  sponsor,  in  accordance  with  the  following  rules: 

2.6  Generate  Appraisal 
Results 

2.6.3b  Determine  Maturity 
Level 

4.6.5. a 

(A)  A  maturity  level  for  a  staged  representation  is  achieved  if 
all  process  areas  within  the  level  and  within  each  lower  level 
are  either  “satisfied”  or  “not  applicable.” 

2.6  Generate  Appraisal 
Results 

2.6.3b  Determine  Maturity 
Level 

4.6. 5. b 

(A)  A  maturity  level  for  a  continuous  representation  is 
achieved  if  the  capability  level  profile  is  at  or  above  the  target 
profile  for  all  process  areas  for  that  maturity  level  and  all  lower 
maturity  levels  in  the  equivalent  staging,  excepting  those 
process  areas  that  are  designated  as  “not  applicable.” 

2.6  Generate  Appraisal 
Results 

2.6.3b  Determine  Maturity 
Level 

4.7 

Reporting  Results 

4.7.1 

(ABC)  The  method  shall  require  documenting  and  reporting 
the  appraisal  findings  and/or  ratings  to  the  appraisal  sponsor 
and  to  the  appraised  organization. 

3.1  Deliver  Appraisal 
Results 

3.1.1  Deliver  Final  Findings 

4.7.2 

(A)  The  method  shall  require  the  submission  of  appraisal  data 
required  by  the  CMMI  Steward  for  the  purpose  of  reporting 
aggregated  appraisal  infonnation  to  the  constituent  community. 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.3  Provide  Appropriate 
Feedback  to  CMMI  Steward 

Specific  requirements  for  submission  of  data 
to  the  CMMI  Steward  are  defined  by  the  SEI 
Lead  Appraisal  Program,  as  part  of  SCAMPI 
Lead  Appraiser  training  and  authorization. 

4.7.3 

(  ABC)  The  method  shall  require  that  the  appraisal  record  be 
provided  to  the  appraisal  sponsor  for  retention. 

3.2  Package  and  Archive 
Appraisal  Assets 

3.2.2  Generate  Appraisal 
Record 
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Glossary 


The  MDD  glossary  defines  many,  but  not  all,  terms  used  in  this  document.  The  following  additional 
sources  for  terms  and  definitions  should  be  considered  supplementary  to  the  MDD  glossary: 

•  CMMI  model  glossary  and  terminology 

•  ARC  glossary 

Terms  that  are  particularly  significant  to  this  document  are  duplicated  from  the  model 
document  or  ARC  for  convenience. 

affirmation  An  oral  or  written  statement  confirming  or  supporting 

implementation  (or  lack  of  implementation)  of  a  CMMI  model 
specific  practice  or  generic  practice.  Affirmations  are  usually 
provided  by  the  implementers  of  the  practice  and/or  internal  or 
external  customers,  but  may  also  include  other  stakeholders  (e.g., 
managers,  suppliers).  (See  “oral  affirmation”  and  “written 

affirmation.”)  [derived  from  MDD  method  overview] 


alternative  practice  A  practice  that  is  a  substitute  for  one  or  more  generic  or  specific 

practices  contained  in  CMMI  models  that  achieves  an  equivalent 
effect  toward  satisfying  the  generic  or  specific  goal  associated  with 
model  practices.  Alternative  practices  are  not  necessarily  one-for- 
one  replacements  for  the  generic  or  specific  practices,  [arc  vi.i  and 

CMMI  model  glossary] 


appraisal  An  examination  of  one  or  more  processes  by  a  trained  team  of 

professionals  using  an  appraisal  reference  model  as  the  basis  for 
determining,  as  a  minimum,  strengths  and  weaknesses,  [arc  vi.2] 

A  summary  statement  describing  the  ratings  generated  as  outputs 
of  the  appraisal,  and  the  conditions  and  constraints  under  which 
the  appraisal  was  performed.  The  ADS  should  be  used  for  public 
disclosures  of  maturity  level  or  capability  level  ratings  so  they  can 
be  interpreted  accurately,  [local] 

appraisal  findings  The  results  of  an  appraisal  that  identify  the  most  important  issues, 

problems,  or  opportunities  for  process  improvement  within  the 
appraisal  scope.  Appraisal  findings  are  inferences  drawn  from 
corroborated  objective  evidence,  [cmmi  model  glossary  and  arc  vi.i] 


Appraisal 
Disclosure 
Statement  (ADS) 
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appraisal  input 

appraisal  method 
class 


appraisal  modes  of 
usage 


appraisal 

objectives 

appraisal  output 

appraisal 

participants 

appraisal 

rating 


appraisal  record 


appraisal 
reference  model 

appraisal  scope 


appraisal  sponsor 


The  collection  of  appraisal  information  required  before  data 
collection  can  commence,  [iso  98C  and  arc  vi.t] 

A  family  of  appraisal  methods  that  satisfy  a  defined  subset  of 
requirements  in  the  Appraisal  Requirements  for  CMMI  (ARC). 
These  classes  are  defined  so  as  to  align  with  typical  usage  modes  of 
appraisal  methods,  [derived  from  ARC  vl.O,  ARCvl.l] 

The  contexts  in  which  an  appraisal  method  might  be  utilized. 
Appraisal  modes  of  usage  identified  for  the  SC  AMPI  method 
include  internal  process  improvement,  supplier  selection,  and 
process  monitoring. 

The  desired  outcome(s)  of  an  appraisal  process,  [arc  vi.i] 


All  of  the  tangible  results  from  an  appraisal  (see  “appraisal 
record”).  [ISO  98C  and  ARC  vl.l] 

Members  of  the  organizational  unit  who  participate  in  providing 
information  during  the  appraisal.  [Cmmi  model  glossary  and  arc  vl.l] 

The  value  assigned  by  an  appraisal  team  to  (a)  a  CMMI  goal  or 
process  area,  (b)  the  capability  level  of  a  process  area,  or  (c)  the 
maturity  level  of  an  organizational  unit.  The  rating  is  determined 
by  enacting  the  defined  rating  process  for  the  appraisal  method 
being  employed.  [CMMI  model  glossary  and  ARC  vl.l] 

An  orderly,  documented  collection  of  information  that  is  pertinent 
to  the  appraisal  and  adds  to  the  understanding  and  verification  of 
the  appraisal  findings  and  ratings  generated,  [derived  from  iso  98C  and 
ARCvl.l] 

The  CMMI  model  to  which  an  appraisal  team  correlates 
implemented  process  activities,  [cmmi  model  glossary  and  arc  vi.i] 

The  definition  of  the  boundaries  of  the  appraisal  encompassing  the 
organizational  limits  and  the  CMMI  model  limits  within  which  the 
processes  to  be  investigated  operate,  [derived  from  CMMI  model  glossary, 
ISO  98C  and  ARC  vl.l] 

The  individual,  internal  or  external  to  the  organization  being  appraised, 
who  requires  the  appraisal  to  be  performed,  and  provides  financial  or 
other  resources  to  carry  it  out.  [derived  from  iso  98C  and  ARCvl.l] 


111-48 


CMU/SEI-2006-HB-002 


appraisal  tailoring 


appraisal 
team  leader 

artifact 


assessment 


capability 

evaluation 


CMMI  Steward 


consensus 


consolidation 


Selection  of  options  within  the  appraisal  method  for  use  in  a 
specific  instance.  The  intent  of  tailoring  is  to  assist  an  organization 
in  aligning  application  of  the  method  with  its  business  needs  and 

objectives.  [CMMI  model  glossary  and  ARC  vl.l] 


The  person  who  leads  the  activities  of  an  appraisal  and  has 
satisfied  the  qualification  criteria  for  experience,  knowledge,  and 
skills  defined  by  the  appraisal  method,  [arc  vl.l] 

A  tangible  form  of  objective  evidence  indicative  of  work  being 
performed  that  is  a  direct  or  indirect  result  of  implementing  a  CMMI 
model  practice.  (See  “direct  artifact”  and  “indirect  artifact.”) 

An  appraisal  that  an  organization  does  internally  for  the  purposes 
of  process  improvement.  The  word  assessment  is  also  used  in  the 
CMMI  Product  Suite  in  an  everyday  English  sense  (e.g.,  risk 
assessment).  [ARCvi.i] 

An  appraisal  by  a  trained  team  of  professionals  used  as  a 
discriminator  to  select  suppliers,  to  monitor  suppliers  against  the 
contract,  or  to  determine  and  enforce  incentives.  Evaluations  are 
used  to  gain  insight  into  the  process  capability  of  a  supplier 
organization  and  are  intended  to  help  decision  makers  make  better 
acquisition  decisions,  improve  subcontractor  performance,  and 
provide  insight  to  a  purchasing  organization,  [arc  vl.l] 

The  Software  Engineering  Institute  (SEI)  is  the  Steward  of  the 
CMMI  Product  Suite.  The  CMMI  Steward  supports  and  facilitates 
the  maintenance  and  evolution  of  the  CMMI  Product  Suite. 

A  method  of  decision  making  that  allows  team  members  to 
develop  a  common  basis  of  understanding  and  develop  general 
agreement  concerning  a  decision  that  all  team  members  are  willing 
tO  Support.  [ARC  vl.l] 

The  activity  of  collecting  and  summarizing  the  information 
provided  into  a  manageable  set  of  data  to  (a)  determine  the  extent 
to  which  the  data  are  corroborated  and  cover  the  areas  being 
investigated,  (b)  determine  the  data’s  sufficiency  for  making 
judgments,  and  (c)  revise  the  data-gathering  plan  as  necessary  to 
achieve  this  sufficiency.  [ARCvi.i] 
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corroboration 


The  activity  of  considering  multiple  pieces  of  objective  evidence 
in  support  of  a  judgment  regarding  an  individual  CMMI  model 
practice.  [ARCvi.2] 


coverage  criteria 

data  collection 
session 

direct  artifact 


discovery-based 

appraisal 


document 


findings 


focus  project 


The  specific  criterion  that  must  be  satisfied  in  order  for  coverage 
to  be  claimed.  [ARCvi.i] 

An  activity  during  which  objective  evidence  is  gathered.  Data 
collection  sessions  (or  activities)  include  document  reviews  and 
interviews.  [ARCvi.i a] 

The  tangible  outputs  resulting  directly  from  implementation  of  a 
specific  or  generic  practice.  An  integral  part  of  verifying  practice 
implementation.  May  be  explicitly  stated  or  implied  by  the  practice 
statement  or  associated  informative  material,  [mdd  method  overview] 

An  appraisal  in  which  limited  objective  evidence  is  provided  by 
the  appraised  organization  prior  to  the  appraisal,  and  the  appraisal 
team  must  probe  and  uncover  a  majority  of  the  objective  evidence 
necessary  to  obtain  sufficient  coverage  of  CMMI  model  practices. 
Discovery-based  appraisals  typically  involve  substantially  greater 
appraisal  team  effort  than  verification-based  appraisals,  in  which 
much  of  the  objective  evidence  is  provided  by  the  appraised 
organization.  (See  “verification-based  appraisal”  for  contrast.) 

A  collection  of  data,  regardless  of  the  medium  on  which  it  is 
recorded,  that  generally  has  permanence  and  can  be  read  by 
humans  or  machines,  [arc  vi.i]  Documents  can  be  work  products 
reflecting  the  implementation  of  one  or  more  model  practices. 
These  documents  typically  include  work  products  such  as 
organizational  policies,  procedures,  and  implementation-level 
work  products.  Documents  may  be  available  in  hardcopy,  softcopy, 
or  accessible  via  hyperlinks  in  a  Web-based  environment,  [derived 

from  MDD  method  overview] 


The  conclusions  of  an  assessment,  evaluation,  audit,  or  review  that 
identify  the  most  important  issues,  problems,  or  opportunities 
within  the  appraisal  scope.  Findings  include,  at  a  minimum, 
weaknesses  based  on  corroborated  objective  evidence,  [arc  vi.2] 

A  project  that  provides  objective  evidence  for  every  process  area 
within  the  model  scope  of  the  appraisal  which  addresses  practices 
applicable  to  that  project,  [mdd  vi.2] 
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focused 

investigation 


fully  implemented 

(FI) 


indirect  artifact 


instantiation 


institutionalization 


instruments 


internal  process 
improvement  (IPI) 


A  technique  to  prioritize  appraisal  team  effort  based  on  the 
continuous  collection  and  consolidation  of  appraisal  data,  and 
monitoring  of  progress  toward  achieving  sufficient  coverage  of 
CMMI  model  practices.  Appraisal  resources  are  targeted  toward 
those  areas  for  which  further  investigation  is  needed  to  collect 
additional  data  or  verify  the  collected  set  of  objective  evidence. 

[derived  from  MDD  method  overview] 


A  practice  characterization  value  assigned  to  a  process 
instantiation  when  (1)  one  or  more  direct  artifacts  are  present  and 
judged  to  be  adequate,  (2)  at  least  one  indirect  artifact  and/or 
affirmation  exists  to  confirm  the  implementation,  and  (3)  no 
weaknesses  are  noted,  [mdd  2.4..2] 

An  artifact  that  is  a  consequence  of  performing  a  specific  or 
generic  practice  or  that  substantiates  its  implementation,  but  which 
is  not  the  purpose  for  which  the  practice  is  performed.  This 
indicator  type  is  especially  useful  when  there  may  be  doubts  about 
whether  the  intent  of  the  practice  has  been  met  (e.g.,  a  work 
product  exists  but  there  is  no  indication  of  where  it  came  from, 
who  worked  to  develop  it,  or  how  it  is  used),  [mdd  method  overview] 

The  implementation  of  a  model  practice  used  in  the  appropriate 
context  within  the  boundaries  of  an  organizational  unit,  [mdd  vi.2] 

The  ingrained  way  of  doing  business  that  an  organization  follows 
routinely  as  part  of  its  corporate  culture.  [CMMivi.2] 

Artifacts  used  in  an  appraisal  for  the  collection  and  presentation  of 
data  (e.g.,  questionnaires,  organizational  unit  information  packets). 
[arc  vi.i]  In  SCAMPI,  instruments  can  be  used  to  collect  written 
information  relative  to  the  organizational  unit’s  implementation  of 
CMMI  model  practices.  This  can  include  assets  such  as 
questionnaires,  surveys,  or  an  organizational  mapping  of  CMMI 
model  practices  to  its  corresponding  processes. 

An  appraisal  mode  of  usage  in  which  organizations  appraise 
internal  processes,  generally  to  either  baseline  their  process 
capability,  to  establish  or  update  a  process  improvement  program, 
or  to  measure  progress  in  implementing  such  a  program,  [derived  from 

MDD  method  overview] 
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A  meeting  of  appraisal  team  members  with  appraisal  participants 
for  the  purpose  of  gathering  information  relative  to  work  processes 
in  place,  [arc  vi.i]  In  SCAMPI,  this  includes  face-to-face 
interaction  with  those  implementing  or  using  the  processes  within 
the  organizational  unit.  Interviews  are  typically  held  with  various 
groups  or  individuals,  such  as  project  leaders,  managers,  and 
practitioners.  A  combination  of  formal  and  informal  interviews 
may  be  held  and  interview  scripts  or  exploratory  questions 
developed  to  elicit  the  information  needed. 

A  practice  characterization  value  assigned  to  a  process 
instantiation  when  (1)  one  or  more  direct  artifacts  are  present  and 
judged  to  be  adequate,  (2)  at  least  one  indirect  artifact  and/or 
affirmation  exists  to  confirm  the  implementation,  and  (3)  one  or 
more  weaknesses  are  noted,  [mdd  2.4.2] 

A  person  who  has  achieved  recognition  from  an  authorizing  body 
to  perform  as  an  appraisal  team  leader  for  a  particular  appraisal 
method.  [ARCvi.i] 

A  subset  of  the  appraisal  team  members,  typically  two  or  three, 
assigned  primary  responsibility  for  collection  of  sufficient 
appraisal  data  to  ensure  coverage  of  their  assigned  reference  model 
process  areas  or  projects,  [local] 

A  project  that  provides  objective  evidence  for  one  or  more  process 
areas  within  the  model  scope  of  the  appraisal  which  address 
practices  performed  by  projects,  [mdd  vi.2] 

A  practice  characterization  value  assigned  when  the  appraisal  team 
determines  insufficient  objective  evidence  exists  to  state  that  the 
practice  is  implemented.  That  is,  direct  artifacts  are  absent  or 
judged  to  be  inadequate,  no  other  evidence  (indirect  artifacts, 
affirmations)  supports  the  practice  implementation,  and  one  or 
more  weaknesses  are  noted,  [mdd  2.4.2] 


A  practice  characterization  value  assigned  when  the  project  has  not 
yet  reached  the  phase  in  the  lifecycle  within  the  appraisal  scope  to 
have  implemented  the  practice,  [mdd  2.4.2] 
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Documents  or  interview  results  used  as  indicators  of  the 
implementation  or  institutionalization  of  model  practices.  Sources 
of  objective  evidence  can  include  instruments,  presentations, 
documents,  and  interviews,  [derived  from  mdd] 

Affirmation  in  the  form  of  a  verbal  statement  provided  to  the 
appraisal  team  via  an  interactive  forum  in  which  the  appraisal  team 
has  control  over  the  interaction  (e.g.,  the  ability  to  interrupt,  ask 
questions,  or  redirect  the  discussion  to  other  subjects).  Interview 
responses  (whether  provided  during  a  “live”  session  or  via 
teleconference,  video  teleconference,  or  other  similar  technology) 
are  examples  of  oral  affirmations.  Alternative  forms  of  oral 
affirmations  include  presentations  and  demonstrations  of  a  tool  or 
mechanism  related  to  implementation  of  a  CMMI  model  practice, 
as  long  as  these  presentations  and  demonstrations  are  provided  in 
an  interactive  setting.  (See  “affirmation.”)  [derived  from  mdd  Appendix  B] 

The  collection  of  projects  and  support  functions  that  provides 
instantiations  of  practices  used  within,  and  representative  of,  an 
organizational  unit,  [mdd  vi.2] 

That  part  of  an  organization  that  is  the  subject  of  an  appraisal.  An 
organizational  unit  deploys  one  or  more  processes  that  have  a 
coherent  process  context  and  operates  within  a  coherent  set  of 
business  objectives.  An  organizational  unit  is  typically  part  of  a 
larger  organization,  although  in  a  small  organization,  the 
organizational  unit  may  be  the  whole  organization.  [Derived  from 
CMMI  model  glossary,  ISO  98C  and  ARC  vl.2] 


A  practice  characterization  value  assigned  to  a  process 
instantiation  when  (1)  direct  artifacts  are  absent  or  judged  to  be 
inadequate,  (2)  indirect  artifacts  or  affirmations  suggest  that  some 
aspects  of  the  practice  are  implemented,  and  (3)  one  or  more 
weaknesses  are  noted;  or  (1)  one  or  more  direct  artifact  is  present 
and  judged  to  be  adequate,  (2)  no  other  evidence  (indirect  artifacts 
or  affirmations)  supports  the  direct  artifact(s),  and  (3)  one  or  more 
weaknesses  are  noted,  [mdd  2.4.2] 
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The  assignment  of  a  value  describing  the  extent  to  which  a  CMMI 
model  practice  is  implemented.  It  is  used  as  a  mechanism  to  reach 
appraisal  team  consensus.  The  range  of  values  for  practice 
characterization  values  include  Fully  Implemented  (FI),  Largely 
Implemented  (LI),  Partially  Implemented  (PI),  Not  Implemented 
(NI),  and  Not  Yet  (NY).  Practice  characterization  values  are 
assigned  to  each  CMMI  model  practice  for  each  process 
instantiation  within  the  organizational  scope,  and  aggregated  to  the 
organizational  unit  level,  [local] 

An  objective  attribute  or  characteristic  used  as  a  “footprint”  to 
verify  the  conduct  of  an  activity  or  implementation  of  a  CMMI 
model  specific  or  generic  practice.  Types  of  practice 
implementation  indicators  include  direct  artifacts,  indirect 
artifacts,  and  affirmations,  [derived  from  15504-9  and  MDD  method  overview] 


Findings  created  after  synthesizing  corroborated  objective 
evidence.  Preliminary  findings  are  provided  to  appraisal 
participants  for  validation.  (See  also  “findings.”)  [ARCvi.ia] 

In  SCAMPI,  a  source  of  objective  evidence  that  includes 
information  prepared  by  the  organization  and  delivered  visually  or 
verbally  to  the  appraisal  team  to  aid  in  understanding  the 
organizational  processes  and  implementation  of  CMMI  model 
practices.  This  typically  includes  such  mechanisms  as  orientation 
or  overview  briefings,  and  demonstrations  of  tools  or  capabilities. 

[derived  from  MDD  method  overview] 


The  set  of  factors  documented  in  the  appraisal  input  that  influences 
the  judgment  and  comparability  of  appraisal  ratings.  These  include, 
but  are  not  limited  to,  (a)  the  size  of  the  organizational  unit  to  be 
appraised,  (b)  the  demographics  of  the  organizational  unit,  (c)  the 
application  domain  of  the  products  or  services,  (d)  the  size, 
criticality,  and  complexity  of  the  products  or  services,  and  (e)  the 
quality  characteristics  of  the  products  or  services.  [Cmmi  model  glossary] 
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An  appraisal  mode  of  usage  in  which  appraisals  are  used  to 
monitor  process  implementation  (for  example,  after  contract  award 
by  serving  as  an  input  for  an  incentive/award  fee  decision  or  a  risk 
management  plan).  The  appraisal  results  are  used  to  help  the 
sponsoring  organization  tailor  its  contract  or  process  monitoring 
efforts  by  allowing  it  to  prioritize  efforts  based  on  the  observed 
strengths  and  weaknesses  of  the  organization’s  processes.  This 
usage  mode  focuses  on  a  long-term  teaming  relationship  between 
the  sponsoring  organization  and  the  development  organization 
(buyer  and  supplier),  [derived  from  MDD  method  overview] 


The  set  of  goal  ratings  assigned  to  the  process  areas  in  the  scope  of 
the  appraisal.  In  CMMI,  also  known  as  the  process  area  profile. 

[derived  from  IS098c  and  ARC  vl.l] 


(See  “appraisal  rating.”)  [CMMI  model  glossary  and  ARC  vl.l] 

Rating  given  to  a  goal  when  the  aggregate  of  validated  findings 
does  not  negatively  impact  achievement  of  the  goal.  Rating  given 
to  a  process  area  in  the  staged  representation  of  the  CMMI  model 
when  all  of  its  goals  are  rated  “satisfied.”  [arc  vi.2] 

Exemplary  or  noteworthy  implementation  of  a  CMMI  model 
practice.  [CMMI  model  glossary  and  ARC  vl.l] 


A  determination  that  the  coverage  requirements  have  been  met. 
See  “coverage”  and  “coverage  criteria.”  [arc  vi.i] 

An  appraisal  mode  of  usage  in  which  appraisal  results  are  used  as 
a  high  value  discriminator  to  select  suppliers.  The  results  are  used 
in  characterizing  the  process-related  risk  of  awarding  a  contract  to 

a  Supplier,  [derived  from  MDD  method  overview] 


An  organizational  group  that  provides  objective  evidence  for 
practices  within  the  model  scope  of  the  appraisal  which  address 
organizational  infrastructure  or  functions,  [mdd  vi.2] 

See  “appraisal  tailoring.” 
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An  appraisal  in  which  the  focus  of  the  appraisal  team  is  on 
verifying  the  set  of  objective  evidence  provided  by  the  appraised 
organization  in  advance  of  the  appraisal,  in  order  to  reduce  the 
amount  of  probing  and  discovery  of  objective  evidence  during  the 
appraisal  on-site  period.  (See  “discovery-based  appraisal”  for 
contrast.) 

The  ineffective,  or  lack  of,  implementation  of  one  or  more  CMMI 
model  practices.  [CMMI  model  glossary  and  ARC  vl.l] 

Affirmation  in  the  form  of  a  written  statement  or  document 
provided  to  the  appraisal  team.  Like  an  oral  affirmation,  a  written 
affirmation  is  not  an  output  or  artifact  of  a  process.  Typically,  it  is 
a  written  statement  about  a  process  that  has  been  specifically 
generated  as  clarifying  information  for  an  appraisal  team. 
Comments  or  rationale  statements  about  processes  contained  in 
objective  evidence  mappings,  as  well  as  responses  to 
questionnaires,  can  be  written  affirmations.  Presentation  and 
demonstration  materials  not  provided  in  an  interactive  setting  to 
the  appraisal  team  can  also  be  written  affirmations  if  they  are  not 
outputs  of  the  process,  in  which  case  they  could  be  direct  or 
indirect  artifacts  instead.  (See  “affirmation.”)  [derived  from  mdd 
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